Nginx: Forward HTTPS traffic to a proxy server requiring authentication

I need to set up a transparent HTTP/HTTPS server (proxy X) with NGINX to proxy the traffic with the authorization needed to the proxy endpoint (proxy Y). The proxy endpoint (B) requires basic authorization to accept traffic.

The analogy would be like this:

Client C requests URL (e.g. ifconfig.co) -> Custom DNS resolves the request to proxy X -> Proxy (X) accepts request and manipulate header to add basic authentication -> Proxy X forwards request to proxy endpoint Y

It should achieve the exact result as the below cURL: Here I use https://ifconfig.co to check the endpoint IP.

HTTP

curl -x http://proxy_endpoint:proxy_port -U 'username':'password' -k https://ifconfig.co

HTTPS

curl -x http://proxy_endpoint:proxy_port -U 'username':'password' -k https://ifconfig.co

Testing:

To test the setup, I add a record in my /etc/hosts to resolve ifconfig.co to my Proxy X server

Then, I do a cURL

#http
curl http://ifconfig.co

#https
curl https://ifconfig.co

A successful result should give me the IP of the endpoint proxy Y

HTTP traffic:

I have extensively searched online for a solution using NGINX modules and I have found a very helpful answer here: https://serverfault.com/a/345244/954119

However, this answer only helped me send HTTP traffic, not HTTPS. Here is what I configured for HTTP on port 80 in nginx.conf:

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  300;

    proxy_set_header  Host            $host;
    proxy_set_header  X-Real-IP       $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;

  server {
    listen *:80;
    access_log /var/log/nginx/nginx.access.log;
    error_log /var/log/nginx/nginx.error.log;

    location / {
        proxy_pass http://proxy_endpoint_Y:PORT;
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Proxy-Authorization "Basic base64-encoded-string";
    }
  }


}

So I set the http directive with server that listens on port 80 and configured location with proxy_set_header Proxy-Authorization with bas64 encoded username/pass.

This works very well actually and does proxy HTTP(port 80) traffic to the proxy endpoint, however, HTTPS traffic won't work.

HTTPS traffic

So, when I changed listen to 443 instead:

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;

    keepalive_timeout  300;

  server {
    listen *:443;
    access_log /var/log/nginx/nginx.access.log;
    error_log /var/log/nginx/nginx.error.log;

    location / {
        proxy_pass http://proxy_endpoint_Y:PORT;
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Proxy-Authorization "Basic base64-encoded-string";
    }
  }


}

cURL error on my Client C:

curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

In my access logs, I get a very weird log:

xx.xxx.xx.x - - [26/Mar/2022:21:43:54 +0000] "\x16\x03\x01\x00\xE1\x01\x00\x00\xDD\x03\x03W\x87>\x1E+1\xDFb\x13\x1E,\xC0\xE9\xEF\x07\xAB[\xEA!\xBE\x17\xC23\x8D\xBD\xA4\xEA\xB5\xD5s\x8AO\x00\x00\x5C\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"

I just need to manipulate HTTPS traffic to add authorization header? Is that possible? Or is there another way of approach?

Note:

I've tried using stream module which I used before to send HTTPS traffic to endpoints, but with no authorization, but I don't see any option in NGINX stream module documentation to manipulate request headers. http://nginx.org/en/docs/stream/ngx_stream_upstream_module.html

System Environment:

nginx version: nginx/1.21.6 built by gcc 9.3.0 (Ubuntu 9.3.0-10ubuntu2) built with OpenSSL 1.1.1f 31 Mar 2020 TLS SNI support enabled