To answer my own question:
The following assumes that the nvidia driver is working well enough to edit grub settings. If this is not the case, boot into recovery mode. Now remove existing nvidia drivers:
sudo apt remove --purge nvidia*
Edit the grub config:
sudo nano /etc/default/grub
And edit the following line to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nomodeset"
For nvidia-driver-495 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_psr=0"
seems to work great.
Update grub:
sudo update-grub
Right now you should be able to boot normaly (not with secondary screen), but you can't use your graphics card. So to fix that:
Install nvidia drivers the 'normal' way:
sudo apt install nvidia-driver-xxx
xxx = 470 at time of writing.
Edit the grub config:
sudo nano /etc/default/grub
And edit the following line to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvidia-drm.modeset=1 i915.enable_psr=0"
I am not too sure about i915.enable_psr=0
, but it seems to fix the screen flickering.
Update grub:
sudo update-grub
Edit nvidia config:
sudo nano /etc/gdm3/custom.conf
And uncomment (remove #) from the line:
WaylandEnable=false
To clarify, I'm using 20.04. Later releases may rely more on wayland so the steps might change.