Delaying full crontab execution per user

In a large scale multi tenant environment where user's make heavy use of minutely crons, many crons setup by users will run on the minute or on the hour causing noticeable CPU spikes at the start of each minute/hour.

I'm looking for a way to delay a user's crontab execution by 0-30 seconds (randomised) in order to level out the cpu usage across the minute.

I'm aware you can sleep/delay a specific cron but I am wondering if there is a way to do this to the entire crontab on a per user basis.

Users can't be trusted with randomly sleeping their own jobs on a per job bases, and while user's have access to their crontab and could remove said delay, a script would be put in-place to check and restore the delay functionality. Of course this could still be implemented by a find and replace on a per job basis, however it would be cleaner to do it on a per user crontab basis instead - somewhere a top of the crontab, if possible?

Here's one option:

1. Disable the use of per-user crontabs (using e.g. cron.allow and cron.deny)

2. Require users to place scripts to run periodically into specific locations (e.g. $HOME/cron/minutely, $HOME/cron/hourly, etc).

3. Introduce a system-level script that runs jobs from those directories on a specified schedule. This script can implement randomized start times to prevent the thundering herd problem.

This solves your problem at the expense of some flexibility, since users can no longer create cron jobs with arbitrary schedules.

Your "system-level script that runs jobs" might be as simple as appropriately configured systemd timer units, which already support this sort of randomization via the RandomizedDelaySec configuration option. Setting this up for a multi-tenant solution might require some creativity.