Score:0

Constant power outages, wasting time to determine the last executed command in my .sh script. What command to use to record progress?

in flag

(I found a perfect solution by using Google Forms and a simple curl command, I wrote it as an answer if anyone ever needs it)

I execute a long .sh script with thousands of commands and once a power outage occurs, I need a lot of time to determine which was the last executed command so I can resume working. I need a creative solution first regarding where the log would be the best to be stored (locally, or on the net, google spreadsheet maybe), and second - a command that I will place on strategic lines in the script and whose sole purpose is to create a descriptive output in the log i.e. if I see 1-SVK in the log, I will be able to resume working within a minute.

So what is the best place to store such a log (non-sensitive data) for a start ? And what command should I use ? Only the output of the most recent execution of that command should be logged, I will modify the command properly and place it on each strategic line in the script so it will provide a correct output in the log (ex: 1-SVK, 2-SVK).

The code in the bash script should look like this:

[command that will print 1-SVK somewhere]
curl command 1
curl command 2
curl command 3
[command that will print 2-SVK somewhere]
curl command 4
curl command 5
curl command 6
...

Thank you in advance for your valuable advice

Emre Talha avatar
ar flag
Adding a logger would be better for interruptions, but if there are constant power outages, I would recommend buying a UPS (uninterrupted power supply). Constantly cutting the power of a computer is not safe for the components and the data inside the disks.
Elio avatar
in flag
Already considering that, it's just UPSs are rather expensive and they're not quite the stuff you get with a battery for a laptop for an example. Whatever I saw in stores would give me 5-6 minutes tops, depending on power consumption of the device of course (which is low). Maybe I am wrong and I should consult an expert for UPSs. I simply consider a curl command that will post an entry to a google spreadsheet, but I just need something more elegant because unfortunately I am not able to recreate the POST request for making an entry in a cell in G Spreadsheet.
raj avatar
cn flag
raj
There is a very old solution that was used in data centers when the admins needed to have absolutely sure storage for system logs. Print the output you need to a printer :)
Elio avatar
in flag
Sure raj, but don't forget that in 2022 the paper got extremely expensive :') I know because I have a job in a Public Procurement sector. But still, around 150 papers / day would be an overkill :))
Score:0
as flag

There are many ways to solve this problem, and the best solution depends on the specific details of the situation. However, some general advice would be to use a command like 'tee' to send the output of the script to both stdout and a log file. This way, you will have a record of the script's output even if the power goes out. Another option would be to use a tool like 'screen' or 'tmux' to keep the script running in the background even if the terminal is closed. This way, you can reconnect to the script and see the last few lines of output even if the power goes out.

Elio avatar
in flag
Thank you for your input. The output I get for a single curl command is huge, which is why I must use --output /dev/null on each curl command. And I can run up to 10,000 curl commands within a day. That will cause many other issues by creating a huge log file, assuming it will keep logging and create history of all logs. I am sorry, how will anything keep running in the background during a power outage ? My PC will be turned off.
Mohamed Elgazar avatar
as flag
There are a few ways to keep a script running in the background even if the terminal is closed. One option is to use the 'screen' or 'tmux' command. This will allow you to reconnect to the script and see the last few lines of output even if the power goes out. Another option is to use a tool like 'nohup' to keep the script running even if the terminal is closed.
Levente avatar
cn flag
I would assume both `screen` and `nohup` make sense only if the operating system remains up and running. I however had the impression that OP's power outages result in reboots.
Elio avatar
in flag
@Levente You are guessing right. Power outage = PC shuts down, as I replied earlier to Mr. Mohamed.
Score:0
in flag

Here is one perfect solution I just found, I am more than satisfied with this, I tested it and works great.

https://johnmu.com/using-curl-to-add-rows-to-google/

Leaving my question alive in case it would help someone in the future.

curl 'https://docs.google.com/forms/u/0/d/e/[ID_OF_YOUR_FORM]/formResponse' --data 'entry.[NUMERICAL_ID_OF_THE_ENTRY]=[THE_TEXT_YOU_WANT_TO_SEE_AS_AN_ANSWER_IN_THE_FORM]' --silent >/dev/null
Levente avatar
cn flag
From the command it is not entirely obvious: how are you getting authenticated in order to be authorized to update the contents of the document?
Elio avatar
in flag
You don't need any authentication, you are actually guest-providing an answer in a public google form. And the answers in the form are very tidy, simple html, and chronologically sorted.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.