Score:0

Error uploading large files (>2gb) through nginx reverse proxy to container

za flag

I am running an application in a docker container (https://github.com/filebrowser/filebrowser/) that I am trying to upload very large (>2) gb files to. The container exposes the app on a nonstandard port on localhost which I connect to through a reverse proxy (so I can run multiple apps on one machine). The problem is when I upload a file over 2097152 bytes it errors out. I can see that the files is fully uploaded to nginx and that it partially makes it to the contianerized app but then hangs for a long time at 2097152 bytes before erroring out causing the containerized app to give a unexpected EOF error. I suspect it's something wrong with nginx becuase when I upload a file to the containerized app directly it works fine. I've tried adding a variety of directives to the nginx config and am stumped. My nginx config for the site is:

server {
    server_name example.com;
    client_max_body_size    30g;
    proxy_buffer_size       1024k;
    proxy_buffers 4 1024k;
    proxy_busy_buffers_size 1024k;
    proxy_max_temp_file_size 10000m;
    proxy_connect_timeout   10000;
    proxy_send_timeout      10000;
    proxy_read_timeout      10000;
    send_timeout            10000;
    client_header_timeout   10000;
    client_body_timeout     10000;
    
    location / {
    proxy_pass http://localhost:8088/;
    proxy_http_version 1.1;
    proxy_cache_bypass $http_upgrade;

    proxy_set_header Upgrade           $http_upgrade;
    proxy_set_header Connection        "upgrade";
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_set_header X-Forwarded-Port  $server_port;

    proxy_connect_timeout              10000s;
    proxy_send_timeout                 10000s;
    proxy_read_timeout                 10000s;
    }}

The error I get in nginx is:

2022/04/14 20:26:56 [error] 3286524#3286524: *13184 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: example.com, request: "POST /api/resources/Fedora-KDE-Live-x86_64-35-1.2.iso?override=false HTTP/1.1", upstream: "http://[::1]:8088/api/resources/Fedora-KDE-Live-x86_64-35-1.2.iso?override=false", host: "example.com", referrer: "https://example.com/files/" 

Edit: After doing lots of digging, it seems that the solution was to remove the proxy_buffer_size, proxy_buffers, and proxy_busy_buffer_size and add proxy_request_buffer no; I don't know why this works but it does.

hu flag
Thanks for asking and sharing part of your solution. When you write "remove the proxy_buffer_size", do you mean that you are using the defaults? or set it to 0? Please share the actual config lines you are using, thanks in advance! I have a similar problem, nginx is out-of-memory (oom) killed due to spiking in memory use at the end of big uploads.
za flag
@klausthorn When I say I "removed proxy_buffer_size" I meant I commented out those lines in the config and used the default values. I'm not sure what those values are but I'm running nginx/1.18.0 on Raspberry Pi OS 64 bit. Hopefully this helps with your problem.
Score:3
cn flag
nh2

Simple answer:


All the other options apply only to downloads to the client (responses). Quoting from the docs:

  • proxy_buffer_size

    ... size of the buffer used for reading the first part of the response received from the proxied server ...

  • proxy_buffers

    ... buffers used for reading a response from the proxied server ...

  • proxy_busy_buffers_size

    ... When buffering of responses from the proxied server ...

Score:0
hu flag

I also hit the 2GB limit. proxy_request_buffering off; did not work for me. Adding the two options below to the / location is the only thing that did work for me:

proxy_buffering      off;
client_max_body_size 0;
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.