I am using multiple Compute Engine Instances and randomly sometimes they seem to be loosing network access. Nothing changed, the certain instance in this case was running about 80 days without any error.
In /var/log/syslog the following erros appear:
Jun 2 17:15:11 alpha google_guest_agent[451]: ERROR main.go:178 Error watching metadata: Get "http://169.254.169.254/computeMetadata/v1//?recursive=true&alt=j>
Jun 2 17:17:07 alpha vnstatd[457]: Warning: Writing cached data to database took 107.0 seconds.
Jun 2 17:17:50 alpha dhclient[355]: DHCPREQUEST for XXX on ens4 to 169.254.169.254 port 67
Jun 2 17:18:32 alpha dhclient[355]: DHCPACK of XXX from 169.254.169.254
Jun 2 17:20:37 alpha systemd[1]: Starting GCE Workload Certificate refresh...
Jun 2 17:22:32 alpha vnstatd[457]: Warning: Writing cached data to database took 119.8 seconds.
Jun 2 17:23:18 alpha OSConfigAgent[1177944]: 2023-06-02T17:21:31.4637Z OSConfigAgent Error main.go:231: Get "http://169.254.169.254/computeMetadata/v1/?recurs>
Jun 2 17:28:27 alpha vnstatd[457]: Warning: Writing cached data to database took 166.0 seconds.
Jun 2 17:32:25 alpha CRON[2851637]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jun 2 17:39:26 alpha OSConfigAgent[1177944]: 2023-06-02T17:38:23.9547Z OSConfigAgent Error main.go:231: Get "http://169.254.169.254/computeMetadata/v1/?recurs>
Jun 2 17:40:28 alpha vnstatd[457]: Warning: Writing cached data to database took 561.0 seconds.
Jun 2 17:40:59 alpha vnstatd[457]: Warning: Writing cached data to database took 50.0 seconds.
Jun 2 17:41:39 alpha dhclient[355]: bound to XXX -- renewal in 73 seconds.
Jun 2 17:43:02 alpha dhclient[355]: DHCPREQUEST for XXX on ens4 to 169.254.169.254 port 67
Jun 2 17:43:43 alpha dhclient[355]: DHCPACK of XXX from 169.254.169.254
Jun 2 17:45:04 alpha vnstatd[457]: Warning: Writing cached data to database took 237.0 seconds.
Jun 2 17:53:23 alpha vnstatd[457]: Warning: Writing cached data to database took 462.3 seconds.
Jun 2 17:54:49 alpha vnstatd[457]: Warning: Writing cached data to database took 63.3 seconds.
Jun 2 17:58:09 alpha OSConfigAgent[1177944]: 2023-06-02T17:57:05.2089Z OSConfigAgent Error main.go:231: Get "http://169.254.169.254/computeMetadata/v1/?recurs>
Jun 2 18:03:45 alpha vnstatd[457]: Warning: Writing cached data to database took 469.3 seconds.
Jun 2 18:04:50 alpha dhclient[355]: bound to XXX -- renewal in 413 seconds.
Jun 2 18:07:40 alpha vnstatd[457]: Warning: Writing cached data to database took 47.5 seconds.
Jun 2 18:11:10 alpha OSConfigAgent[1177944]: 2023-06-02T18:09:28.2124Z OSConfigAgent Warning: Error waiting for task (attempt 1 of 10): rpc error: code = Unav>
Jun 2 18:11:57 alpha dhclient[355]: DHCPREQUEST for XXX on ens4 to 169.254.169.254 port 67
At the same time disk write speed is at max for the whole time. After about 2 hours I could again connect via SSH and everything worked normal again. I can also access the local metadata server (169.254.169.254) without any problems.
The instances are all running Debian 11.
Do you think vnstat is causing this?
