others-how to solve the docker nginx 111: Connection refused problem?

Nov 10, 2022

1. Purpose

In this post, I will demonstrate how to solve the following error when using nginx in docker:

2022/11/09 09:05:41 [error] 63#63: *32 connect() failed (111: Connection refused) while connecting to upstream, client: 10.13.4.3, server: test.bswen.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "test.bswen.com"
2022/11/09 09:05:41 [warn] 63#63: *32 upstream server temporarily disabled while connecting to upstream, client: 10.13.4.3, server: test.bswen.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "test.bswen.com"
2022/11/09 09:05:41 [error] 63#63: *32 connect() failed (111: Connection refused) while connecting to upstream, client: 10.13.4.3, server: test.bswen.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "test.bswen.com"
2022/11/09 09:05:41 [warn] 63#63: *32 upstream server temporarily disabled while connecting to upstream, client: 10.13.4.3, server: test.bswen.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "test.bswen.com"
10.13.4.3 - - [09/Nov/2022:09:05:41 +0000] "GET / HTTP/1.1" 502 497 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15" "-"
2022/11/09 09:10:38 [error] 63#63: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 152.89.196.211, server: test.bswen.com, request: "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1", upstream: "http://127.0.0.1:3000/?XDEBUG_SESSION_START=phpstorm", host: "test.bswen.com:80"
2022/11/09 09:10:38 [warn] 63#63: *35 upstream server temporarily disabled while connecting to upstream, client: 152.89.196.211, server: test.bswen.com, request: "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1", upstream: "http://127.0.0.1:3000/?XDEBUG_SESSION_START=phpstorm", host: "test.bswen.com:80"
152.89.196.211 - - [09/Nov/2022:09:10:38 +0000] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1" 502 497 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" "-"
2022/11/09 09:10:38 [error] 63#63: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 152.89.196.211, server: test.bswen.com, request: "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1", upstream: "http://127.0.0.1:3000/?XDEBUG_SESSION_START=phpstorm", host: "test.bswen.com:80"
2022/11/09 09:10:38 [warn] 63#63: *35 upstream server temporarily disabled while connecting to upstream, client: 152.89.196.211, server: test.bswen.com, request: "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1", upstream: "http://127.0.0.1:3000/?XDEBUG_SESSION_START=phpstorm", host: "test.bswen.com:80"
^C

The core error message is:

connect() failed (111: Connection refused) while connecting to upstream

The nginx configuration is as follows:


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    upstream yapi_up {
        server 127.0.0.1:3000;
    }

    include /etc/nginx/conf.d/*.conf;
}

server {
    listen       80;
    listen  [::]:80;
    server_name  test.bswen.com localhost;


    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://yapi_up;
    }

The deployment of nginx and the service on the host is as follows:

There is a web service listening on 127.0.0.1:3000
The nginx is deployed in a docker container, and using port mapping from container’s port 80 to host’s port 80
The nginx is using proxy_pass function to proxy the incoming request from outside the container to the local web service

But when we tried to access from the host’s ip address, we got this error:

2022/11/09 09:10:38 [error] 63#63: *35 connect() failed (111: Connection refused) while connecting to upstream, client: 152.89.196.211, server: test.bswen.com, request: "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1", upstream: "http://127.0.0.1:3000/?XDEBUG_SESSION_START=phpstorm", host: "test.bswen.com:80"
2022/11/09 09:10:38 [warn] 63#63: *35 upstream server temporarily disabled while connecting to upstream, client: 152.89.196.211, server: test.bswen.com, request: "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1", upstream: "http://127.0.0.1:3000/?XDEBUG_SESSION_START=phpstorm", host: "test.bswen.com:80"
^C

Test the local web service’s availability:

[root@mx ~]# curl http://localhost:3000 -vvv
* About to connect() to localhost port 3000 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 3000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:3000
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 1746
< Last-Modified: Wed, 09 Nov 2022 06:53:05 GMT
< Cache-Control: max-age=0
< Content-Type: text/html; charset=utf-8
< Date: Thu, 10 Nov 2022 06:00:48 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
<!DOCTYPE html>
<html>
<head>
...

It’s running fine.

It seems that the local web service that is listening on port 3000 is refusing to let the client connect to it. why?

2. Solution

Solution #1

The above problem can be depicted by the following diagram:

The nginx in a docker container is just trying to access the upstream service using the following url:

proxy_pass http://127.0.0.1:3000

And the docker container of nginx is started as follows:

docker container run \
  --rm \
  --name mynginx1 \
  --volume "$PWD/conf":/etc/nginx \
  -p 0.0.0.0:80:80 \
  -d \
  nginx

You can see that the nginx is listening on port 80 inside the container, which is mapped to the host’s port 80.

Actually, the connection from nginx to upstream is just happened inside the docker container, which using a special network. So ,the real connection is as following diagram shows: The connection is proxied inside the container, does not reach the outside.

By default , the docker container is a bridge network using docker0 as the bridge from container to host and other containers.

[root@mx nginx]# ip addr show docker0
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:72:69:75:17 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:72ff:fe69:7517/64 scope link
       valid_lft forever preferred_lft forever
[root@mx nginx]

Then we can change the upstream url inside the nginx configuration from 127.0.0.1 to the docker0’s address:

proxy_pass http://172.17.0.1:3000

This solution can be described as following diagram:

However, under Windows and macOS platforms, there is no docker0 virtual network card. At this time, you can use the special DNS name host.docker.internal to resolve the host IP.

proxy_pass http://host.docker.internal:3000

Solution #2

Instead of using the bridge network, we can choose the host network for nginx container.

In Docker, a host is a machine that runs one or more containers. Docker network host, also known as Docker host networking, is a networking mode in which a Docker container shares its network namespace with the host machine.

User-defined bridge networks are best when you need multiple containers to communicate on the same Docker host. Host networks are best when the network stack should not be isolated from the Docker host, but you want other aspects of the container to be isolated.

We can start our nginx using docker as follows:

docker run -d --name mynginx1 --network host nginx

Notice that we do not use -p for port mapping again, because every port is bound to the host’s network interface, e.g. If there is a port 80 listening inside container, then there is a port 80 listening on the host.

Now we can change the nginx configuration as follows:

proxy_pass http://127.0.0.1:3000

Using the host network does not require modifying nginx.conf, and localhost can still be used, so the versatility is better than the previous method. However, since the isolation of the host network is not as good as that of the bridge network, the security of using the host network is not as high as that of the bridge network.

3. Summary

In this post, I demonstrated how to solve the 111: Connection refused problem when using nginx in docker, the key point is to understand the network model of docker . That’s it, thanks for your reading.