Score:1

S3 through Nginx Proxy doesn't allow more than 10 files to be downloaded simultaneously

ar flag

I setup a Nginx Proxy in front of my S3 object storage and I'm able to download files without any issues so far.

However, if I try to download the same file from, let's say, 30 different servers, at the same time, only the first 10-11 servers are able to continue downloading while the rest gets cancelled with the following error;

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

This is the command I'm running to download the file;

sudo curl -f -Ls https://cdn.mydomain.com/files/somefile.tar.gz | tar xvz

And this is my nginx config file under /etc/nginx/sites-enabled/cdn.conf;

server {
    listen 80;
    server_name cdn.mydomain.com;

    if ($host != "cdn.mydomain.com") {
        return 444;
    }

    access_log /var/log/nginx/reverse-access.log;
    error_log /var/log/nginx/reverse-error.log;

    location /files {
        proxy_pass https://myBucket.s3.nl-ams.scw.cloud/files;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

My nginx.conf is the default file that gets generated as I first installed the nginx on my server.

I don't know what's limiting this but how can I allow more concurrent downloads in this setup?

Michael Hampton avatar
cz flag
Check your logs.
Marry Jane avatar
ar flag
@MichaelHampton I just checked and there are bunch of `no space left on device` errors but I'm not sure how this is related to my disk and I have about 13GB of space left.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.