Join Log in
Score:0
Simon Richard avatar Server

NGINX double proxy not allowing websocket connections

lb flag
Simon Richard

I have two nginx proxies setup on my machine, one to unwrapp SSL and the other to do application-specific proxying (only the second one is version controlled). When I only had one proxy I was able to make successful websocket connections, but after moving to two all websocket upgrade requests are responded to with a 502 Bad Gateway error. I can confirm that normal http/https requests are working with my double proxy setup. Here's my current config.

Proxy 1

server {

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Note: You should disable gzip for SSL traffic.
        # See: https://bugs.debian.org/773332
        #
        # Read up on ssl_ciphers to ensure a secure configuration.
        # See: https://bugs.debian.org/765782
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        server_name staging.ambitx.io;

        location / {
                proxy_pass http://127.0.0.1:81;
                include proxy_params;
        }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/staging.ambitx.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/staging.ambitx.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = staging.ambitx.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen 80 default_server;
        listen [::]:80 default_server;

        server_name staging.ambitx.io;
    return 404; # managed by Certbot

}

proxy_params

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Proxy 2

(running on docker with port 81 on the host machine bound to port 80 on the container)

resolver 127.0.0.11 ipv6=off;

server {
  listen       80;
  listen  [::]:80;

  location / {
    root /var/www/staticfiles;
    index  index.html index.htm;
    try_files $uri /index.html =404;
  }
  
  location /ws {
    access_log off;

    proxy_pass       http://wsserver;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }

  location /api {
    proxy_pass http://apiserver;
  }
}

I originally had the statements below in the location /ws block of Proxy 2's config...

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

but I removed them because they would be overwriting the headers set by Proxy 1.

Any ideas? Please let me know if you need more information.

20
0 + 0
Score:0
Ivan Shatsky avatar Server
gr flag
Ivan Shatsky

Of course you should not drop the WebSocket proxying support from any of your intermediate proxy servers. WebSocket proxying is a very special task, to allow protocol switching and keep WebSocket connection established and alive you should return its support to your first nginx proxy as well:

...
location / {
    proxy_pass http://127.0.0.1:81;
    include proxy_params;
}
location /ws {
    proxy_pass http://127.0.0.1:81;
    include proxy_params;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}
...

Removing all the three proxy_set_header directives from your second nginx proxy was also a bad idea. While X-Real-IP and X-Forwarded-For headers will be passed to your WebSocket app exactly as being set by your first proxy, the Host header is a special one. Unless being set explicitly, it will be passed equal to the upstream name used in the proxy_pass directive, i.e. Host: wsserver. As you can read from the proxy_set_header directive documentation:

By default, only two fields are redefined:

proxy_set_header Host $proxy_host;
proxy_set_header Connection close;

(the second one will surely break any attempt to establish a WebSocket connection as well). So to keep the original Host header from the client request (usually a good idea, you can read more about this header here), return the

proxy_pass Host $http_host;

line to both location / { ... } and location /ws { ... } of your second nginx proxy configuration.

0 + 0
Simon Richard avatar
lb flag
Simon Richard
This worked for me, thanks. For the purpose of keeping proxy 1 agnostic of proxy 2 (since proxy 1 isn't currently version controlled), I decided to use the "more sophisticated example" (the second one) listed on this website: https://nginx.org/en/docs/http/websocket.html instead and just add it to the `location / { ... }` block in proxy 1. That way if I change the websocket url but forget about proxy 1 it won't break. The only downside to this that I can think of is that it won't be able to handle http/2.0, but my server wasn't setup to handle it anyways.
0
Reply
Ivan Shatsky avatar
gr flag
Ivan Shatsky
That's sound reasonable, I prefer those way of proxying WebSocket too. About HTTP/2, you may found [this](https://serverfault.com/questions/765258/use-http-2-0-between-nginx-reverse-proxy-and-backend-webserver) thread quite interesting.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.