Ubuntu 22.04 Jammy - cron seems to be ignoring timezone

I have a server configured to use the Pacific/Auckland timezone, however cron is running in UTC.

In the previous OS version this was sufficient to get everything to operate using a specified timezone (including cron):

sudo ln -sf /usr/share/zoneinfo/$TIMEZONE /etc/localtime
echo "$TIMEZONE" | sudo tee /etc/timezone
sudo dpkg-reconfigure --frontend noninteractive tzdata
sudo timedatectl set-timezone $TIMEZONE

The above seems to work everywhere except for cron - both date and timedatectl return what is expected, but cron still runs in UTC i.e.

$ date
Wed Jul  5 12:50:01 NZST 2023

$ timedatectl
               Local time: Wed 2023-07-05 12:50:01 NZST
           Universal time: Wed 2023-07-05 00:50:01 UTC
                 RTC time: Wed 2023-07-05 00:50:00
                Time zone: Pacific/Auckland (NZST, +1200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Adding something like this to the crontab 57 0 * * * date >> /tmp/cron-test.log results in output of Wed Jul 5 12:57:01 NZST 2023 being written to the log file. My understanding is that cron is supposed to run in local time, however that is not what I am observing.

I have read several StackOverflow threads and have tried the following to no avail:

• Setting TZ and CRON_TZ in /etc/crontab
• Setting TZ and CRON_TZ in crontab -e
• Setting TZ and CRON_TZ in /etc/environment
• Setting TZ and CRON_TZ in /etc/systemd/system/cron.service.d/override.conf - syslog says it ignores this
• Setting TZ and CRON_TZ in Service/Environment in /etc/systemd/system/cron.service.d/override.conf

After each attempt I ran sudo systemctl daemon-reload && sudo service cron restart.

My observations there were that the only effect setting TZ had in any context was to change the timezone that the commands in the crontab ran in, not cron itself. Setting CRON_TZ had no effect on anything.

Also I have tried this env -i $(cat /proc/$(</var/run/crond.pid)/environ|xargs -0 echo) date +%Z in an attempt to see what timezone cron is running in and it returns NZST which is what it should be running in, but unfortunately that does not seem to be the case...

I found this thread where someone was seeing pretty much the same behaviour: https://ubuntuforums.org/showthread.php?t=2477727 - for him updating packages and rebooting seems to have resolved the issue, however this doesn't seem to have worked for me.

Any wisdom here would be appreciated.

UPDATE 1

This post cron takes UTC time instead of local time has an answer that seems to apply in my scenario:

If you haven't set the hardware time to match system time, then cron's going to work off hardware time

Running sudo timedatectl set-local-rtc 1 seems to make cron run at the right time for a moment, but outputs the below and then gets reset to UTC after a while.

timedatectl status
               Local time: Wed 2023-07-05 14:13:01 NZST
           Universal time: Wed 2023-07-05 02:13:01 UTC
                 RTC time: Wed 2023-07-05 14:13:01
                Time zone: Pacific/Auckland (NZST, +1200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: yes

Warning: The system is configured to read the RTC time in the local time zone.
         This mode cannot be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.

So it seems that cron is running on hardware time. Unfortunately setting the hardware clock to local time is not recommended and it gets undone anyway. Sadly this is not a solution, but does open another avenue for investigation.

UPDATE 2

Investigation into hardware clock lead me to a couple of findings:

1. You can set it to use local time a couple of different ways - sudo hwclock --systohc --localtime and timedatectl set-local-rtc 1
2. I found this page where someone discusses legacy approaches and the /etc/adjtime file.

It turns out that timedatectl and hwclock both update the /etc/adjtime file and not in the same way. Setting it local time using either mechanism make timedatectl status complain. However running hwclock --systohc --localtime prior to timedatectl set-local-rtc 0 seems to retain some settings that allow cron to run at the correct time and prevents timedatectl status from complaining.

At this point cron is running correctly, however I am unsure if the settings will stick after an NTP update. I will try rebuilding some servers and answer the question if it fixes it.

As per this answer by warren - the issue is that cron was using the hardware clock not the system clock.

I found that doing the following resolved it:

# use local time for hardware time and set drift factor
hwclock --systohc --localtime

# use UTC time for hardware time 
timedatectl set-local-rtc 0

This makes the hardware clock use local time and creates a config file at /etc/adjtime then reverts the hardware clock back to UTC, but leaves the config file in place. Note that timedatectl set-local-rtc 1 does not set the config params that are require for this to work so hwclock seems to be required.

The /etc/adjtime config created by the above solution seems to make cron run in the right timezone without breaking things like NTP updates.

It may be this is not the best solution, but it does solve the problem.