Join Log in
Score:2
eugenio_chisari avatar Server

Large file download from self hosted website

cn flag
eugenio_chisari

Sorry if the question is simple, I am new to web development and self-hosted servers.

We have a self hosted website, which is supposed to have a button to download a large zip file (1 GB). For this, we have a simple solution in the index.html file:

<form action="path/to/file.zip" style="display: inline;">
    <button type="submit" class="btn btn-light">DOWNLOAD</button>
</form>

This usually worked fine. The problem is, whenever multiple visitors try to download the file at the same time, the server runs out of RAM and the website crashes. This is because every time someone clicks on the download button, the file appears to be loaded on RAM during the whole download process. When multiple visitors click the Download button, the file is loaded on RAM multiple times. As additional info, the website is built with python+Flask, and the server has 12 GB of RAM.

I have seen that in this answer (How to stop Apache from crashing my entire server?) they suggest to "take load off Apache for long-running processes", but I am not sure how to achieve that. Is there any solution you can suggest to solve this problem?

Thank you very much in advance.

88
1 + 7
vn flag
ceejayoz
Consider hosting the file somewhere off-server, like Amazon S3.
2
Reply
br flag
Chopper3
That doesn't sound right at all - what actual webserver engine are you using?
1
Reply
djdomi avatar
za flag
djdomi
even file file would have 50gb ore more it should stream it instead of opening
1
Reply
vn flag
ceejayoz
@Chopper3 I'd bet the requests are going through the app's front controller rather than direct to the webserver.
2
Reply
eugenio_chisari avatar
cn flag
eugenio_chisari
@Chopper3 we use Nginx. @ djdomi could you point me to some resource explaining how to do that? Thank you all.
1
Reply
djdomi avatar
za flag
djdomi
@eugenio_chisati if you using plane nginx, post the nginx conf, this is not a normal behavior for nginx, you could try to use `directio 50m` to verify if your still running ooom
0
Reply
eugenio_chisari avatar
cn flag
eugenio_chisari
Hello everyone, thank you very much for your feedback, it was very helpful for me to be able to ask around the right questions and find an answer. It turns out, our nginx server was simply forwarding the requests from the public address to the local address in our server; the app was then actually using a Tornado web server. So the solution was simply to use nginx directly to serve the static files, and only forwarding to the tornado server for the non-static part. This solved the memory problem. Thank you again for the help and for putting up with my beginner questions :)
1
Reply
Score:1
eugenio_chisari avatar Server
cn flag
eugenio_chisari

thank you very much for your feedback, it was very helpful for me to be able to ask around the right questions and find an answer.

It turns out, our nginx server was simply forwarding the requests from the public address to the local address in our server; the app was then actually using a Tornado web server. So the solution was simply to use nginx directly to serve the static files, and only forwarding to the tornado server for the non-static part. This solved the memory problem.

Thank you again for the help and for putting up with my beginner questions :)

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.