nginx denies all requests to a virtual host . requests are coming from a nginx tcp forwarder

Summary - need to add whitelist ips to the mysite1.example.com. now when they are added it doesn't work as every request is coming as originated from the load balancer server.

Im working on a setup with a front seating Nginx host with a upstream backend to loadbalance all tcp packets on port 443 to backend servers.

nginx config of Loadbalancer server running nginx - server C is as below

stream {
        upstream stream_backend {
                hash $remote_addr consistent;
                server 10.15.15.3:443;   ## server A
                server 10.15.15.9:443;   ## server B
        }


        server {
                listen     443;
                proxy_pass stream_backend;
                proxy_timeout 5s;
                proxy_connect_timeout 5s;
        }
}

server A and server B has below nginx.conf. they are identical servers with apps.

it has two virtual hosts running in each. they are working fine.

http {

    server {
        server_name mysite1.example.com;
        listen *:443 ssl;
        listen [::]:443 ssl;
        
        allow 123.45.85.220; # this seems not working
        deny all; # only this is working

        
        location ^~ /static/ {
            ...
        }
        ...
        
        ssl_certificate        file.pem;
        ssl_certificate_key    file.key;
    }


    server {
        server_name mysite2.example.com;
        listen *:443 ssl;
        listen [::]:443 ssl;
        
        
        location /somethin {
            ...
        }
        location /something2{
            ...
        }
        
        ssl_certificate        file.pem;
        ssl_certificate_key    file.key;
    }
}

what I need is to whitelist only few ips to the virtual host mysite1.example.com. the issue I face is that the nginx running on Server A and B see the load balancer Ip as the client Ip. so when tried adding allow IP; deny all. doesn't work for any host as it has the load balancer IP on all requests as the client IP.

Can someone guide me on adding proxy IP configs to achieve the above mentioned setup running fine. Setup is complete except for the IP whitelist issue.

p.s SSl termination happens at the back-end servers , server A and Server B

I've searched through the web and found these helpful but still couldn't figure out how to get it all working.

https://stackoverflow.com/questions/40873393/nginx-real-client-ip-to-tcp-stream-backend https://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/

My approach would be:

Filter in the load balancer using SNI:

Basically: SNI allows the proxy to see the domain name, without terminating TLS. Here is a guide. And here are the nginx docs to combine it into a solution. If you can do this, you can then deny based upon $ssl_preread_server_name.

if ( $ssl_preread_server_name ~* mysite1.example.com && $remote_addr !~* <whitelisted_ip> ) {
           return 404;

The real_ip module also seems like a good solution.

Edit

The PROXY protocol would actually be better than real_ip. But this would still mean, that your backend servers get hit by requests that they'd end up blocking. The SNI solution catches them at the load-balancer.

seems like your proxy not passing the real IP try to add this parameter

proxy_set_header X-Real-IP $remote_addr;

can you check the access logs which ip there ?