Join Log in
Score:1
Lostsoul avatar Ubuntu

Is there a way to queue scripts in ubuntu/debian/linux?

ph flag
Lostsoul

I have a strange question(at least to me!), but I'm not sure how to even approach it.

Background

I have a few batch jobs(python scripts called via cron or python script being called by bash script). For example, 20 ETL jobs and 20 data processing jobs. That occurs at different times.

Current approach

I keep my server on and run cron jobs to run the scripts at their intended time(they do not all run at the same time..one may run at 1pm whereas another runs at 5pm). This is working perfectly right now

Problem I want to save money and the jobs do not have to start exactly at the time I set, they could be a few minutes or an hour or two off so I was thinking of using preemptible instances that start the server on a schedule.

I was thinking of starting the server every hour and then it processes any jobs that are past their schedule that has not been done yet.

I have no idea how to do this! Is there a way I can schedule this within ubuntu? I thought of setting up an external DB and have a script that queries it to see what processes need to start but I felt it was overkill, I suspect this maybe a common problem that has a simpler solution.

Please let me know!

102
0 + 0
user535733 avatar
cn flag
user535733
What do you mean by "*starting the server every hour*"? Is this server on bare metal? Or is it a cloud instance? Or container or VM on some host?
0
Reply
Lostsoul avatar
ph flag
Lostsoul
@user535733 On GCP, you can schedule a server to startup at a certain time. Normally I'd just run the script in the startup but using that approach i'd have to have like 40 servers with different startup scripts. I'm trying to have one server do different things based on a queue/time window.
0
Reply
pLumo avatar
in flag
pLumo
You can use `@reboot` as cron interval. Then check within a script what needs to be done
0
Reply
Lostsoul avatar
ph flag
Lostsoul
@pLumo I apologize I am not familiar with that command. I searched it but I found the common to reboot my server? Do you know what I should search?
0
Reply
user535733 avatar
cn flag
user535733
For those who are not big on cloud-speak, GCP = Google Cloud. This sounds like a normal task for systemd timers or anacron, not cron. Both timers and anacron are designed to run if the system has missed a job due to inactivity. You don't need to start the machine every hour for missed jobs. You can start it once a day...or once a month. Start it when you actually need it, and let timers or anacron handle the logic of figuring out what was missed and which jobs to run and in which order.
0
Reply
bac0n avatar
cn flag
bac0n
think you are looking for [OnUnitActiveSec=](https://unix.stackexchange.com/questions/235102/systemd-timer-onunitactivesec-vs-oncalendar-for-anacron-replacement/235303#235303)
0
Reply
bac0n avatar
cn flag
bac0n
Related: [Unison and systemd timer problem](https://askubuntu.com/q/656075/986805)
0
Reply
Lostsoul avatar
ph flag
Lostsoul
Wow thank you @user535733 and everyone for your help! I looked up Anacron and it looks like what I need. If you want to put the answer I can upvote/accept. It was hard to find info on this when I searched so having it as a answer would hopefully help others with this similar challenge. Thank you!
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.