why does systemd tag my self-made service "thawing" after running it for a few days?

I have found the strange tag of "thawing" running my self-made service using systemctl status XXX like this:

On the contrary, I found it running several threads via htop:

At first I thought it was a python problem, until I found a golang binary encountered the same problem:

[root]# systemctl status auto_quote
● auto_quote.service - run run_quotelive.sh ## <-change run_XXX.sh
   Loaded: loaded (/usr/lib/systemd/system/auto_quote.service; enabled; vendor preset: disabled)
   Active: active (running) (thawing) since Wed 2023-06-21 15:05:51 CST; 1 months 18 days ago
 Main PID: 1765370 (bash)
    Tasks: 8 (limit: 5366)
   Memory: 56.5M
   CGroup: /system.slice/auto_quote.service
           ├─1765370 /bin/bash /root/run_quotelive.sh ## <-change run_XXX.sh
           └─1765372 ./quotelive

[root]# cat /lib/systemd/system/auto_quote.service
[Unit]
Description=run run_quotelive.sh ## <-change run_XXX.sh
After=network.service


[Service]
Type=simple
ExecStart=/bin/bash /root/run_quotelive.sh ## <-change run_XXX.sh
Restart=on-failure
RestartSec=3
## systemd bears daemon duty

[Install]
WantedBy=multi-user.target

The run_quotelive.sh bash script that auto_quote.service runs is this:

# substitute PATH with the desired Abusolute path of the program, binary or script
# substitute COMMAND with the shell command to run the program, binary or script
# i.e. binary/executable named as XXX, change it to ./XXX
# i.e. python3 script named as XXX.py, change it to python3 XXX.py
# the script below would run it at background and leave starting/failing time
# as well as program output( both normal print and errors) into PATH/out.log
cd /root/quotelive
cmd='./quotelive'
newcmd="$cmd &>> out.log"
echo -e '\n'  `date +'%Y/%m/%d %a, %X'`: STARTING - "$cmd" '\n'  >> out.log
eval $newcmd
echo -e '\n'  `date +'%Y/%m/%d %a, %X'`: PROCESS FAILED  >> out.log
exit 5 # make an error exit for service to restart

So is there anything wrong with my edition of auto_quote.service or run_quotelive.sh?

I changed the script with the chance to exit peacefully:

cd /root/quotelive
cmd='./quotelive'
echo -e '\n'  `date +'%Y/%m/%d %a, %X'`: STARTING - "$cmd" '\n'  >> out.log
$cmd >>out.log 2>&1
if [ $? -ne 0]; then
  echo -e '\n'  `date +'%Y/%m/%d %a, %X'`: PROCESS FAILED  >> out.log
  exit 5 # make an error exit for service to restart
fi

For testing purpose you just change /root/quotelive to any directory you want, and ./quotelive to python3 -m http.server 12345, which is also a server that serves forever.

But it keeps on posting thawing. I have also noticed the printout from the program quotelive supposed to be redirected to /root/quotelive/out.log also appeared in systemctl status auto_quote.

Its not clear what quotelive should do, but it obviously exit often with your exit 5 directive. Thawing means systemd thinks its not a good idea to restart failing service eternally because if it fails always it should be fixed and not waste resources, so after few restarts it is thawed.

First problem you have is that you run background job and then you make exit 5 making systemd think it failed however its not. With that you get forever created instances of quotelive.

If quotelive program is intended to run forever you should use:

# substitute PATH with the desired Abusolute path of the program, binary or script
# substitute COMMAND with the shell command to run the program, binary or script
# i.e. binary/executable named as XXX, change it to ./XXX
# i.e. python3 script named as XXX.py, change it to python3 XXX.py
# the script below would run it at background and leave starting/failing time
# as well as program output( both normal print and errors) into PATH/out.log
cd /root/quotelive
cmd='./quotelive'
newcmd="$cmd >> out.log"
echo -e '\n'  `date +'%Y/%m/%d %a, %X'`: STARTING - "$cmd" '\n'  >> out.log
eval $newcmd && exit 0
echo -e '\n'  `date +'%Y/%m/%d %a, %X'`: PROCESS FAILED  >> out.log
exit 5 # make an error exit for service to restart

If it should run and do job and then restarted: use Restart=Always in service file.

Although, this wrapper script looks odd, you may consider running quotelive from systemd directly.