timedatectl on ubuntu server 20.04

I have an ubuntu server (20.04) where timedatectl is failing to set the timezone. On the first call, it always fails, on the second it thinks it has succeeded but actually it hasn't:

Beforehand the timezone is Europe/Paris:

root@oxpad:~# timedatectl 
               Local time: Wed 2021-08-11 02:02:44 CEST
           Universal time: Wed 2021-08-11 00:02:44 UTC 
                 RTC time: Wed 2021-08-11 00:02:44     
                Time zone: Europe/Paris (CEST, +0200)  
System clock synchronized: yes                         
              NTP service: active                      
          RTC in local TZ: no                          

** The time is CEST == Central European Summer Time **

root@oxpad:~# date
Wed Aug 11 02:02:46 CEST 2021

Change the timezone:

root@oxpad:~# timedatectl set-timezone Europe/London
Failed to set time zone: Access denied
root@oxpad:~# timedatectl set-timezone Europe/London
root@oxpad:~#

Apparent success

root@oxpad:~# timedatectl 
               Local time: Wed 2021-08-11 01:02:58 BST
           Universal time: Wed 2021-08-11 00:02:58 UTC
                 RTC time: Wed 2021-08-11 00:02:58    
                Time zone: Europe/London (BST, +0100) 
System clock synchronized: yes                        
              NTP service: active                     
          RTC in local TZ: no                         

But actual failure

root@oxpad:~# date
Wed Aug 11 02:02:59 CEST 2021
root@oxpad:~# 

and indeed no other services see the updated local time. It seems that timedatectl is alone in thinking it has succeeded.

This is completely repeatable. It is an issue because we will be deploying quite a few of these servers and if the timezone cannot be set, we have a problem. I would really like to understand the relationship between timedatectl and "real" system time, as seen by date and other processes, as that would probably lead to an understanding of the problem. Any pointers would be welcome.

Interestingly, setting the timezone on an Ubuntu 20.04 desktop system with timedatectl works fine. It is almost as if timedatectl sets the timezone over dbus but dbus is not set up properly on the server version of the OS.

All of this testing is on newish installations, not more than a few days old and clean apart from adding a few packages.

Does anyone know what is going on here? Help would be much appreciated.

More logs: On the Ubuntu 20.04 server there is an error in the logs when changing time, on the 20.04 desktop the same service log shows success. The error:

root@oxpad:~# systemctl status systemd-timedated
● systemd-timedated.service - Time & Date Service
     Loaded: loaded (/lib/systemd/system/systemd-timedated.service; static; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-timedated.service(8)
             man:localtime(5)
             https://www.freedesktop.org/wiki/Software/systemd/timedated

Aug 11 02:02:33 oxpad systemd[1]: systemd-timedated.service: Succeeded.
Aug 11 02:02:38 oxpad systemd[1]: Starting Time & Date Service...
Aug 11 02:02:38 oxpad systemd[1]: Started Time & Date Service.
Aug 11 02:02:38 oxpad systemd-timedated[35339]: Failed to set time zone: Permission denied
Aug 11 02:02:53 oxpad systemd-timedated[35339]: Failed to set time zone: Permission denied

I could not duplicate your error on a plain Ubuntu 20.04 image. You should be checking if the command changes /etc/timezone and the /etc/localtime symlink. Also check what the permissions on those files are. If you are running selinux or other security software then that is another possible cause for the error.

root@ubuntu:~# date
Wed Aug 11 16:05:25 UTC 2021
root@ubuntu:~# timedatectl
               Local time: Wed 2021-08-11 16:05:28 UTC
           Universal time: Wed 2021-08-11 16:05:28 UTC
                 RTC time: Wed 2021-08-11 16:05:29
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
root@ubuntu:~# cat /etc/timezone
Etc/UTC
root@ubuntu:~# ls -l /etc/localtime
lrwxrwxrwx 1 root root 29 Aug 11 15:39 /etc/localtime -> ../usr/share/zoneinfo/Etc/UTC


root@ubuntu:~# timedatectl set-timezone Europe/London
root@ubuntu:~# date
Wed Aug 11 17:05:51 BST 2021
root@ubuntu:~# timedatectl
               Local time: Wed 2021-08-11 17:05:59 BST
           Universal time: Wed 2021-08-11 16:05:59 UTC
                 RTC time: Wed 2021-08-11 16:06:00
                Time zone: Europe/London (BST, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
root@ubuntu:~# cat /etc/timezone
Europe/London
root@ubuntu:~# ls -l /etc/localtime
lrwxrwxrwx 1 root root 35 Aug 11 17:05 /etc/localtime -> ../usr/share/zoneinfo/Europe/London

These steps are another way to automate changing the timezone.

1. Update the /etc/localtime symlink
2. Run dpkg-reconfigure --frontend noninteractive tzdata

You can see these steps will update the /etc/timezone file as well.

root@ubuntu:~# date
Wed Aug 11 16:09:30 UTC 2021
root@ubuntu:~# ln -f -s /usr/share/zoneinfo/Europe/London /etc/localtime
root@ubuntu:~# dpkg-reconfigure --frontend noninteractive tzdata

Current default time zone: 'Europe/London'
Local time is now:      Wed Aug 11 17:10:30 BST 2021.
Universal Time is now:  Wed Aug 11 16:10:30 UTC 2021.

root@ubuntu:~# date
Wed Aug 11 17:10:34 BST 2021
root@ubuntu:~# cat /etc/timezone
Europe/London
root@ubuntu:~# ls -l /etc/localtime
lrwxrwxrwx 1 root root 33 Aug 11 17:10 /etc/localtime -> /usr/share/zoneinfo/Europe/London