Ubuntu 23.04 freezing, leading to a login loop - how to investigate?

I have a fresh install of Ubuntu 23.04.

• Used the new installer.
• One 4TB SSD disk with LUKS encryption.

I use the application VideoHubApp, which scans a directory for movies and makes previews. When I leave this running, most often, this leads to a system freeze, the system restarts automatically, and the next login ends with a login loop.

I can get out of the login loop by re-installing ubuntu-desktop.

sudo su -
apt remove -y ubuntu-desktop;
apt update;
apt dist-upgrade;
apt autoremove;
apt clean;
apt install -y --fix-missing ubuntu-desktop; 
reboot

I did this already 4 times and thinking of writing a script for that :D

But I would prefer to know why the system froze. I can't find anything suspicious in the journalctl - at least not what's not there on normal boots.

How can I investigate what is happening and what prevents Ubuntu to let me log in after such freeze?

If that matters, as the last thing before this started happening, I installed FUSE and NodeJS which the VideoHubApp needs (not sure why FUSE).

• Lenovo T540p, 16 GB RAM, 4 GB swap.
• Also froze with Ubuntu 20.04 when running this app, but the login loop did not occur.
• 4K display with 150 % fractional scaling
• Not modifications except installing some dev packages - Docker, VirtualBox, etc.

I have seen this question: Ubuntu 23.04 Fresh Install Login Loop

But that's not my case - I have just /boot and /, and home dir is within the same partition with /.

I got my answer in /var/log/syslog:

2023-07-24T04:00:09.789535+02:00 opers kernel: [ 2446.293431] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-oomd.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/session.slice/dbus.service,task=ffmpeg,pid=21057,uid=1000
2023-07-24T04:00:09.789537+02:00 opers kernel: [ 2446.295850] Out of memory: Killed process 21057 (ffmpeg) total-vm:28821808kB, anon-rss:13241832kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:38296kB oom_score_adj:200
2023-07-24T04:00:09.789549+02:00 opers kernel: [ 2446.609055] rfkill: input handler enabled
2023-07-24T04:00:10.218341+02:00 opers systemd[2611]: dbus.service: A process of this unit has been killed by the OOM killer.
2023-07-24T04:00:10.308724+02:00 opers sh[14704]: Terminated

As I suspected, the system runs out of memory. But I wonder why it doesn't kill the APP which ate it all, and kills dbus.service instead? Or am I not reading the log properly? Either way, Gnome gets killed, too. Which seems a bit weird choice - to kill the entire desktop environment. Can I prevent that? (A matter for another question.)

As a workaround, I increased my swap file size by 16 GB to 20 GB:

sudo swapoff /swap.img 
sudo dd if=/dev/zero of=/swap.img bs=16M count=1024 oflag=append conv=notrunc
sudo mkswap /swap.img 
sudo swapon /swap.img