Using Ubuntu 22.04 in EC2 via CloudFormation, why does cfn-hup never trigger?

Amazon's documentation includes extensive examples of using cfn-hup to automate updates of instances provisioned via CloudFormation. For one of many examples, see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent-New-Instances-CloudFormation.html. These typically end up pointing to examples like this one for cfn-hup iteslf or this one for the CloudWatch agent.

From the latter we get a typical example:

        01_setupCfnHup:
          files:
             '/etc/cfn/cfn-hup.conf':
               content: !Sub |
                 [main]
                 stack=${AWS::StackId}
                 region=${AWS::Region}
                 interval=1
               mode: '000400'
               owner: root
               group: root
             '/etc/cfn/hooks.d/amazon-cloudwatch-agent-auto-reloader.conf':
               content: !Sub |
                 [cfn-auto-reloader-hook]
                 triggers=post.update
                 path=Resources.EC2Instance.Metadata.AWS::CloudFormation::Init.02_config-amazon-cloudwatch-agent
                 action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource EC2Instance --region ${AWS::Region} --configsets UpdateEnvironment
                 runas=root
               mode: '000400'
               owner: root
               group: root
             "/lib/systemd/system/cfn-hup.service":
                content: !Sub |
                  [Unit]
                  Description=cfn-hup daemon
                  [Service]
                  Type=simple
                  ExecStart=/opt/aws/bin/cfn-hup
                  Restart=always
                  [Install]
                  WantedBy=multi-user.target
          commands:
            01enable_cfn_hup:
              command: !Sub |
                systemctl enable cfn-hup.service
            02start_cfn_hup:
              command: !Sub |
                systemctl start cfn-hup.service

This is used in combination with a UserData script that runs pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz.

Using these examples, cfn-hup never triggers cfn-init when I update my CloudFormation stack. In /var/log/cfn-hup.log I see harmless-looking lines like

2023-07-08 09:49:56,112 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.eu-west-2.amazonaws.com
2023-07-08 09:49:56,115 [INFO] No umask value specified in config file. Using the default one: 0o22

This looks like it's running okay, so why is it not detecting any stack updates?

If I run cfn-hup --no-daemon manually from the shell, my updates are correctly detected and cfn-init is triggered as expected.

(This presumably applies to any OS which uses Python 3.8+, but Ubuntu is the only distribution popular on EC2 which ships with a Python 3.8+ at time of writing.)

First, the eagle-eyed will notice that there are some mixed references to /opt/aws and /usr/local. In fact /usr/local is where the AWS tools get installed, so references to /opt/aws need to be updated - but this doesn't solve the problem.

The issue is that cfn-hup in daemon mode doesn't support the version of Python3 in Ubuntu 22.04. Running cfn-hup (without --no-daemon) manually from the shell produces the stack trace described in that ticket.

The solution is to change the cfn-hup.service definition to schedule it to run periodically in non-daemon mode. For example:

            /etc/cfn/cfn-hup.conf:
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}
                interval=1
                umask=022
              mode: '000400'
              owner: root
              group: root
            # Add whatever hooks are desired here
            /lib/systemd/system/cfn-hup.service:
              content: !Sub |
                [Unit]
                Description=cfn-hup
                [Service]
                Type=oneshot
                ExecStart=/usr/local/bin/cfn-hup -c /etc/cfn --no-daemon
                [Install]
                WantedBy=multi-user.target
              mode: '000400'
              owner: root
              group: root
            /lib/systemd/system/cfn-hup.timer:
              content: !Sub |
                [Unit]
                Description=Run cfn-hup every minute
                [Timer]
                OnCalendar=*-*-* *:*:00
                [Install]
                WantedBy=timers.target
              mode: '000400'
              owner: root
              group: root
          commands:
            01enable_cfn_hup:
              command: systemctl daemon-reload && systemctl enable --now cfn-hup.timer