A really strange confluence of dependencies led to an odd database crash. Context:
- CentOS 7.9
- LVM2-2.02
- Postgresql 12, with data volume on an LVM volume, XFS formatted
- systemd
- dbus
While the system was up and stable, and the database running as normal, I performed a yum update. During the update, several volumes were unmounted, including the one the database was mounted on. This resulted in the panic message:
2021-08-12 21:12:35.418 CEST PANIC: could not open file ""/var/lib/pgsql/current/data/global/pg_control"": No such file or directory
The journald logs shows quite a few packages had been updated in the minutes preceding this crash. There are no intermediary log messages from postgresql which might narrow it down.
Apparently something in /etc/fstab is incorrectly configured, but the following message can be seen long before the yum update:
Aug 12 18:27:32 git.uibk.ac.at systemd-fstab-generator[19273]: Checking was requested for "none", but it is not a device.
and frequently during the update. So I consider this a red herring.
Systemd is updated and restarted:
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Reexecuting.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
That's 45 seconds before the crash. Immediately after this, we see:
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: [/etc/systemd/system/rsyslog.service.d/partof-journald.conf:2] Assignment outside of section. Ignoring.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopped target Remote File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping Remote File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Unset automount mnt-git\x2drepodata.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping mnt-git\x2drepodata.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Set up automount mnt-git\x2drepodata.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Reached target Remote File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Unset automount Arbitrary Executable File Formats File System Automount Point.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping Arbitrary Executable File Formats File System Automount Point.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopped target Local File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping Local File Systems.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Unset automount var-lib-pgsql.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Stopping var-lib-pgsql.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: var-lib-pgsql.automount: Directory /var/lib/pgsql to mount over is not empty, mounting anyway.
Why.... WHY???? .... WHYYYYYYYYYY!!!! At any rate, it appears not to have worked properly.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Set up automount var-lib-pgsql.automount.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Mounting /var/lib/pgsql/current/pg_stat_tmp...
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Mounted /var/lib/pgsql/current/pg_stat_tmp.
Aug 12 21:11:46 git.uibk.ac.at systemd[1]: Reached target Local File Systems.
This volume was also not mounted soon after, and had to be remounted by hand during the recovery process.
A few other messages appear before the PANIC:
Aug 12 21:12:32 git.uibk.ac.at yum[12642]: Installed: kernel-3.10.0-1160.36.2.el7.x86_64
Aug 12 21:12:33 git.uibk.ac.at yum[12642]: Updated: 2:vim-enhanced-7.4.629-8.el7_9.x86_64
Aug 12 21:12:33 git.uibk.ac.at systemd[1]: Reloading.
Aug 12 21:12:34 git.uibk.ac.at yum[12642]: Updated: 7:lvm2-2.02.187-6.el7_9.5.x86_64
Aug 12 21:12:34 git.uibk.ac.at yum[12642]: Updated: 1:nfs-utils-1.3.0-0.68.el7.1.x86_64
And then these two, strangely, not in the right order:
Aug 12 21:12:35 git.uibk.ac.at systemd[1]: postgresql-12.service: main process exited, code=exited, status=1/FAILURE
Aug 12 21:12:35 git.uibk.ac.at kernel: XFS (dm-6): Unmounting Filesystem
dm-6 is currently (after reboot) the swap volume, so ...?
Here's a list of the volumes, excluding the usual suspects:
/dev/mapper/vgdata-git--db on /var/lib/pgsql type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgroot-lvroot on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgroot-lvtmp on /tmp type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgroot-lvvar on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/vda1 on /boot type ext4 (rw,relatime,seclabel,data=ordered)
nfsshare:/git/qt-gitlab on /mnt/git-repodata type nfs (rw,relatime,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountport=635,mountproto=udp,local_lock=none)
none on /var/lib/pgsql/12/pg_stat_tmp type tmpfs (rw,relatime,seclabel,size=16384k,mode=770,uid=102,gid=26)
systemd-1 on /mnt/git-repodata type autofs (rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13188)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13146)
systemd-1 on /var/lib/pgsql type autofs (rw,relatime,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13753)
I've never seen this happen before on any modern Linux system. Is this "expected behavior" of systemd when it is updated or re-executed?
