NGINX long delays when completing proxied WebDAV uploads

I have an internal IIS server running WebDAV being used directly for file uploads and downloads from an android application.

My internal DNS resolves the https://webdav.mydomain.com directly to IIS (bypassing NGINX) and this internal communication appears to work without issue. Upload and download speed are not fantastic for the available wireless speeds, but are what I have grown accustomed to in my IIS configuration (whether due to limits of my hardware or simply IIS WebDAV limitations).

External to the network the URL resolves to the public IP of my NGINX server. When using the application remotely, download speeds also seem acceptable (~25 MB/s).

Upload speeds however are slow. They are about half of the download speeds. However, more important than the slowness is that there is a very long delay at the end of the upload before completing.

My client shows an upload status which reports the # of bytes being uploaded and the total upload size. When an upload reaches about 99% of the total upload size it just sits there for some time before finally completing.

On a 50MB file it will stop around 49MB and wait about 30 seconds before completing. A 3GB upload waited for at least 5 minutes before finally completing successfully.

This issue does not exist at all internally when the NGINX server is not in the mix. I accept that I will have slower throughput externally, and even that the limited NGINX hardware might have some additional throttling. But I am not sure why there is such a long delay in NGINX being able to complete the upload to the IIS server.

Below is the relevant NGINX configuration:

upstream webdav_backend {
        server 10.10.10.102:443;
        keepalive 100;
}

server {
        listen 443 ssl http2;

        server_name webdav.mydomain.com;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Protocol $scheme;

        proxy_redirect off;
        proxy_buffering off;

        proxy_connect_timeout 180s;
        proxy_send_timeout 180s;
        proxy_read_timeout 180s;
        fastcgi_send_timeout 180s;
        fastcgi_read_timeout 180s;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        location @proxy {
                proxy_pass https://webdav_backend;
        }

        location / {
                try_files $uri @proxy;
                client_max_body_size 4G;
        }

        access_log /opt/var/log/nginx/webdav.mydomain.com.remote.log remote_hosts if=$remote_hosts;
        access_log /opt/var/log/nginx/webdav.mydomain.com.access.log;
        error_log /opt/var/log/nginx/webdav.mydomain.com.error.log;
}

So after posting I went back to try an capture some more information.

I noticed that during an upload my IIS server was not receiving data immediately when my upload began. For smaller files, like my 50MB one it seemed that the backend server didn't start receiving the data until it was nearly complete and for larger files I noticed more buffering on the client throughout along with periods of inactivity of the backend server's reception of the data.

I also noticed that the memory on my NGINX server (embedded hardware with low specs) was decreasing about 70MB (75% of available memory left) while the transfer was happening.

This all reinforced to me that the NGINX server was still somehow caching/buffering the data despite "proxy_buffering off" being specified in the configuration. I reviewed the documentation and found:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

Disabling this feature not only resulted in the delays being entirely eliminated, but seems to have improved my upload speed in general about 50%.