Score:2

How can I crontably reboot a Linux host upon network failure?

id flag

I have a personal Ubuntu host, connected to a public/sharing Wi-Fi AP (not under my control).

Sometimes, it would have a network issue, and I am very sure, only restarting the network service would not work. The only way is to reboot it.

My plan is to add a crontab, to test the network connection. If it fails, then reboot the computer.

If I manually run the auto_reboot.sh, it does reboot, when a ping test fail. But running from crontab, it is not working:)

Here is my crontab entry

crontab -l
* * * * * /root/loadrc/transmissionrc/auto_reboot.sh

File /root/loadrc/transmissionrc/auto_reboot.sh

#!/bin/zsh

/root/loadrc/networkrc/ping.sh
rc=$?

if [[ $rc -eq 0 ]]
then
    echo "say The internet is back up."
else
    reboot
fi

File /root/loadrc/networkrc/ping.sh

#!/bin/zsh
((count = 10))                           # Maximum number to try.

while [[ $count -ne 0 ]] ; do
    ping -c 1 8.8.8.8                    # Try once.
    rc=$?
    if [[ $rc -eq 0 ]] ; then
        ((count = 1))                    # If okay, flag loop exit.
    else
        sleep 1                          # Minimise network storm.
    fi
    ((count = count - 1))                # So we don't go forever.
done

exit $rc

I add some logs, and deliberately bring down the Wi-Fi interface:

watch ifconfig wlan0 down
transmissionrc/auto_reboot.sh
#!/bin/zsh

echo "" > /root/loadrc/crontab.log

/root/loadrc/networkrc/ping.sh
rc=$?

if [[ $rc -eq 0 ]]
then
    echo "say The internet is back up."
else
    reboot
fi
networkrc/ping.sh
#!/bin/zsh
((count = 10))                           # Maximum number to try.


while [[ $count -ne 0 ]] ; do
    /usr/bin/ping -c 1 8.8.8.8 >> /root/loadrc/crontab.log 2>&1
    echo "step --> 2" >> /root/loadrc/crontab.log
    rc=$?

    if [[ $rc -eq 0 ]] ; then
        echo "step --> 3" >> /root/loadrc/crontab.log
        ((count = 1))                    # If okay, flag loop exit.
    else
        echo "step --> 4" >> /root/loadrc/crontab.log
        sleep 1                          # Minimise network storm.
    fi
    ((count = count - 1))                # So we don't go forever.
done

exit $rc

File /root/loadrc/crontab.log

/usr/bin/ping: connect: Network is unreachable
step --> 2
step --> 3

which means, in the crontab mode, even the ping test fail, the return code is still zero.

So the question comes to: how can I test the network connection in crontab mode?

djdomi avatar
za flag
it looks like an x and y issue, what is the original issue you trying to solve?
sn flag
Re *"a personal Ubuntu host"*: Wouldn't that make it off-topic?
br flag
I wouldn't use `cron` for that, but `watchdog`, which is specifically designed to reboot the local machine if some condition fails, and allows running custom commands as checks.
Score:7
cn flag
Bob
/usr/bin/ping -c 1 8.8.8.8 >> /root/loadrc/crontab.log 2>&1 
echo "step --> 2" >> /root/loadrc/crontab.log  
rc=$?

I think this checks the exit code of the echo command while your logic needs the exit code of the ping command.

huangyingw avatar
id flag
yes, bug in my echo, thanks for pointing this out.
marcelm avatar
ng flag
@huangyingw So, can you update your question to correct that?
Score:2
il flag

I have three thoughts on possible solutions to your issue:

1. You said:

If I manually run the auto_reboot.sh, it does reboot, when a ping test fail. But running from crontab, it is not working:)

Usually, when a command runs properly in your interactive shell (from the CLI), but fails to run properly under cron it is due to a difference in the environment; e.g. cron has a different PATH than you do from your interactive shell. Typically, the cron environment is: PATH=/usr/bin:/bin. Any script you run run under cron will not be able to find executables that are not on the PATH.

As an aside, you can check the cron environment on your system by simply running env using your crontab:

* * * * * /usr/bin/env > /my/cronlog/location/mycronenvironment.txt 2>&1

In your auto_reboot.sh, you failed to use a full path specifications for reboot. As reboot is typically found in /sbin/reboot, and /sbin may not be in the PATH used by cron, this is a potential problem.

Consequently, I will suggest that you verify the environment (PATH) used by cron, and double-check all of your commands are either: 1) on the cron PATH, or 2) use a full path specification.

2. You run everything out of the /root directory

Ordinarily, /root isn't used for user scripts. Perhaps you are using sudo? Or, perhaps you have done an su to become root? If this is the case, I would comment that this is not best practice, even though it can still work. I feel best practice is to use sudo from your user account for any privilege escalation you need.

Without trying to be pedantic, I'd like to say that the root account has a crontab that runs independently of any user crontab. Also, the root crontab does not require sudo be used - everything done in the root crontab is done with root privileges.

All of that said, I see the call to reboot in your script - a command that requires root privileges to run. This will work as you've written it only when used in the root crontab. Your question did not indicate whether or not you are using su or sudo, and so I went through this in an effort to make two points clear:

  1. If your cron job requires root privileges, it may be best to run that job from the root crontab. The alternative is to use sudo in a user crontab which is potentially awkward if authentication is required for sudo - as is often the case.
  1. The root crontab may be accessed from a regular user account simply by using sudo crontab -e; i.e. one is not required to su to root to access the root crontab.

3. You may have a logical error in your script

As pointed out in another answer, it is not clear that you can rely on the value of rc from your ping.sh script as a condition for reboot. Unfortunately, whether or not this is an issue is masked by what seems to be two different versions of the ping.sh script in your question - it is unclear whether you are using the first version:

#!/bin/zsh
((count = 10))                           # Maximum number to try.

while [[ $count -ne 0 ]] ; do
    ping -c 1 8.8.8.8                    # Try once.
    rc=$?

or the second version:

#!/bin/zsh
((count = 10))                           # Maximum number to try.


while [[ $count -ne 0 ]] ; do
    /usr/bin/ping -c 1 8.8.8.8 >> /root/loadrc/crontab.log 2>&1
    echo "step --> 2" >> /root/loadrc/crontab.log
    rc=$?

Strictly as my personal choice, I would favor combining the code from these two scripts (ping.sh and auto_reboot.sh) into a single script because it seems more straightforward to me, but you may have good reasons for doing it this way, and there's no reason it won't work if done correctly.

huangyingw avatar
id flag
thanks for your answer, yes, I found the reason, I need to use full path of the reboot command. I isolated the logic to a ping.sh, is for better reuse, maybe other of my scripts might need it. it is my personal headless linux just for fun, so, I always logged in as root, the cron and scripts are all run as root. one follow up question: if I(root) not logged in, will the root cron doing its job? what I want is a cron, even no users(including root) is logged in, it would still regularly run, and check the network status, and reboot if needed. @seamus
Seamus avatar
il flag
@huangyingw: *"`one follow up question: if I(root) not logged in, will the root cron doing its job? `"* Yes - `cron` is a *daemon* that runs in the background with no user intervention required; its sole purpose is to run jobs even when the user is not logged in.
huangyingw avatar
id flag
great to hear, thanks. @seamus
Seamus avatar
il flag
@huangyingw: You're welcome - and please don't forget to [vote on any answers you find *useful*, and *select* an answer](https://serverfault.com/help/someone-answers) if appropriate.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.