others-how to solve AccessDeniedException[/usr/share/elasticsearch/data/nodes] when start ElasticSearch using docker?
1. Purpose
In this post, I will show you how to solve the following error when starting ElasticSearch using Docker.
Core error:
{"type": "server", "timestamp": "2023-06-13T08:32:59,414Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "d76a12f01777", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.10.jar:7.17.10]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.10.jar:7.17.10]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:1088) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.10.jar:7.17.10]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:699) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:806) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:792) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.10.jar:7.17.10]",
uncaught exception in thread [main]
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.10.jar:7.17.10]",
"... 6 more"] }
ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438)
at java.base/java.nio.file.Files.createDirectory(Files.java:699)
at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:806)
at java.base/java.nio.file.Files.createDirectories(Files.java:792)
at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300)
at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224)
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298)
at org.elasticsearch.node.Node.<init>(Node.java:429)
at org.elasticsearch.node.Node.<init>(Node.java:309)
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
at org.elasticsearch.cli.Command.main(Command.java:77)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log
2. Solution
2.1 The Environment
- Linux CentOS 7
- Root user
- ElasticSearch 7.17
- Docker 18.09
2.2 The error
When I start an ElasticSearch docker container as follows :
The docker command:
docker run -d -p 9206:9200 -p 9306:9300 -v /home/haystack/es_9206_data:/usr/share/elasticsearch/data --name es206 --restart unless-stopped -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Des.enforce.bootstrap.checks=true" --privileged --user 0 docker.elastic.co/elasticsearch/elasticsearch:7.17.10
Let’s break down the command:
docker run: This command is used to run a Docker container.-d: It runs the container in the background (detached mode).-p 9200:9200 -p 9300:9300: This maps the container’s Elasticsearch ports to the corresponding ports on the host machine. Elasticsearch uses port 9200 for RESTful API access and port 9300 for inter-node communication.-v /home/haystack/es_9206_data:/usr/share/elasticsearch/data: This mounts a host directory to the Elasticsearch container’s data directory. Replace/path/to/host/directorywith the actual path on your host machine where you want to persist the data. This ensures that the data is stored on the host machine and will be retained even if the container is removed or restarted.- `–name es206: This gives a name to the container (optional but recommended).
--restart unless-stopped: This sets the restart policy for the container. It ensures that the container automatically restarts if it stops, unless explicitly stopped by the user.docker.elastic.co/elasticsearch/elasticsearch:7.17.10: This specifies the Elasticsearch Docker image and version to use. In this example, version 7.17.10 is used, but you can replace it with the desired version.
After running this command, Elasticsearch will start in single-node mode, persist its data on the specified host directory, run in the background, and automatically restart if stopped.
But I got the follwing error:
[root@local es_9206_data]# docker run -d -p 9206:9200 -p 9306:9300 -v /home/haystack/es_9206_data:/usr/share/elasticsearch/data --name es206 --restart unless-stopped -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Des.enforce.bootstrap.checks=true" --privileged --user 0 docker.elastic.co/elasticsearch/elasticsearch:7.17.10
d76a12f01777e9c6bcb29f8d3c84e91d46b0c3715de693b9538a74050003a382
[root@local es_9206_data]#
[root@local es_9206_data]# docker logs d7
{"type": "server", "timestamp": "2023-06-13T08:32:57,263Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "d76a12f01777", "message": "version[7.17.10], pid[7], build[default/docker/fecd68e3150eda0c307ab9a9d7557f5d5fd71349/2023-04-23T05:33:18.138275597Z], OS[Linux/3.10.0-1160.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.1/20.0.1+9-29]" }
{"type": "server", "timestamp": "2023-06-13T08:32:57,267Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "d76a12f01777", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2023-06-13T08:32:57,268Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "d76a12f01777", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-13040619671448750029, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Des.enforce.bootstrap.checks=true, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2023-06-13T08:32:59,414Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "d76a12f01777", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.10.jar:7.17.10]",
"at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.10.jar:7.17.10]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:1088) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.10.jar:7.17.10]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:699) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:806) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:792) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.10.jar:7.17.10]",
uncaught exception in thread [main]
"at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.10.jar:7.17.10]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.10.jar:7.17.10]",
"... 6 more"] }
ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438)
at java.base/java.nio.file.Files.createDirectory(Files.java:699)
at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:806)
at java.base/java.nio.file.Files.createDirectories(Files.java:792)
at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300)
at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224)
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298)
at org.elasticsearch.node.Node.<init>(Node.java:429)
at org.elasticsearch.node.Node.<init>(Node.java:309)
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
at org.elasticsearch.cli.Command.main(Command.java:77)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log
The core error is:
ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
2.3 The reasoning
Because the logs indicate that the error is caused by the “Access Permission Missing” problem, I think there is no permission problem inside docker, so the problem must be in the outside “the Host”, for my case, that is the directory : /home/haystack/es_9206_data, Let’s check it:
ls -l /home/haystack/es_9206_data
drwx-xr-x /home/haystack/es_9206_data xxxx
You can find that the directory is exposing limited permissions to groups and others, so we should expand its permissions as follows:
chmod -R 777 /home/haystack/es_9206_data
Then we list it again:
[root@local haystack]# ll
总用量 0
drwxrwxrwx. 4 root root 42 6月 13 16:42 es_9206_data
[root@local haystack]# cd es_9206_data/
[root@local es_9206_data]# ll
总用量 4
drwxrwxrwx. 3 root root 27 6月 13 16:44 data
drwxrwxrwx. 2 root root 4096 6月 13 16:50 logs
2.4 Test again
Then we run the docker command again, we can get this:
[root@local haystack]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19bdbffa8b89 docker.elastic.co/elasticsearch/elasticsearch:7.17.10 "/bin/tini -- /usr/l…" About a minute ago Up 12 seconds 0.0.0.0:9206->9200/tcp, :::9206->9200/tcp, 0.0.0.0:9306->9300/tcp, :::9306->9300/tcp es206
It’s running !!!
3. Summary
In this post, I demonstrated how to solve the ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes]; error when trying to start ElasticSearch in Docker container, the key point is to grant enough permissions to the mapping directory on the host. That’s it, thanks for your reading.