Limit systemd service manual restart rate

dronus

11/21/23, 1:50 PM

I have a system where a service (nginx for example) is sometimes restarted excessively by 'systemd restart nginx' (as result from other scripts, eg. to apply a new configuration or rotate logs).

Even when restarting a service "manually" this way, systemd fails with restart rate limit. Also it is not useful to restart nginx 10 times in a certain second of course.

So how to limit the services manual restart rate? Is there something like a restart-cooldown that would also apply to manual restart requests? The usual settings all apply to automatic restarting by systemd it seems.

295

2 + 0

systemd

systemd-service

Score:0

Server

Abhi

2/23/24, 4:17 PM

What release of operating system are you using? Was nginx package installed using the repository of the Linux distribution or from source code?

Maybe answers to these questions can help determine the possible path needed to fix the issue.

+ 1

dronus

2/26/24, 11:53 AM

This questions should be a comment rather than an answer.

Score:0

Server

A. Darwin

11/21/23, 2:49 PM

You can achieve this with StartLimitIntervalSec and StartLimitBurst.

From the systemd unit man page:

StartLimitIntervalSec=interval, StartLimitBurst=burst

Configure unit start rate limiting. Units which are started more than burst times within an interval time span are not permitted to start any more. Use StartLimitIntervalSec= to configure the checking interval and StartLimitBurst= to configure how many starts per interval are allowed.

interval is a time span with the default unit of seconds, but other units may be specified, see systemd.time(5). Defaults to DefaultStartLimitIntervalSec= in manager configuration file, and may be set to 0 to disable any kind of rate limiting. burst is a number and defaults to DefaultStartLimitBurst= in manager configuration file.

These configuration options are particularly useful in conjunction with the service setting Restart= (see systemd.service(5)); however, they apply to all kinds of starts (including manual), not just those triggered by the Restart= logic.

Note that units which are configured for Restart=, and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually or from a timer or socket at a later point, after the interval has passed. From that point on, the restart logic is activated again. systemctl reset-failed will cause the restart rate counter for a service to be flushed, which is useful if the administrator wants to manually start a unit and the start limit interferes with that. Rate-limiting is enforced after any unit condition checks are executed, and hence unit activations with failing conditions do not count towards the rate limit.

+ 2

dronus

1/11/24, 3:36 PM

As of my experience, it seems that those parameters only affect event-triggered restarts, not manual restarts triggered by 'systemctl restart'. Those manual restarts by systemctl seem to 1) circumvents those limits and 2) still trigger the restart limit. Seems broken by design this way...

dronus

1/11/24, 3:38 PM

Btw. these parameters are fail limits too, they lead to an error response, and not just a limiting of the restart attempts. Anyway, the result is the service is left non-running. So basically manually restart too often -> service NOT running. Would be more useful if the restart would not execute the 'stop', if there where to many restart attempts..

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Limit systemd service manual restart rate

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.