Trying to run 3 commands simultaneously but only 1 of the 3 commands actually functions correctly

MAIN

1/30/24, 12:42 AM

0 */12 * * * /usr/bin/python3 /home/main.py >> file1.txt
0 */12 * * * /usr/bin/python3 /home/main2.py >> file2.txt
0 */12 * * * /usr/bin/python3 /home/main3.py >> file3.txt

This is what I've tried so far. /\

Explanation: Each of these scripts is just the same web scraper script scraping data from 14000 different links from a Google worksheet, each script will scrap a third of the links. I know when each script is running because they'll update a worksheet cell to "On". So when the cronjob starts 3 cells immediately change to "On" but only one section of the worksheet is updating with new data, telling me that only 1 of the 3 scripts is actually working, usually the second script (main2.py). Eventually, the second script stops (Doesn't finish) and the first script starts working. It's outputting no error information so I'm struggling to understand the issue. Also, running the task with only one script works fine but it takes roughly 10 hours to complete without any issues.

TLDR: All of them "Turn on" but only the second command is actually functioning correctly.

I'm doing this on a trial digital ocean ubuntu server FYI, is it possible that the scripts are too demanding for the server?

The only solution I can think of is to create a separate server for each script, but that's probably a bad idea(I don't know a thing about servers)

0 + 1

server

matigo

1/30/24, 1:55 AM

"is it possible that the scripts are too demanding for the server?" ⇠ There is no way anybody can answer this without knowing: (1) the code (2) the target sites (3) the amount of processing that goes on in the scrape process (4) the size of the DO instance. If you believe this to be an issue with system load, using `top` or `glances` while the python scripts are running would be a good way to identify this.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Trying to run 3 commands simultaneously but only 1 of the 3 commands actually functions correctly

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.