Choppy window movement Ubuntu 22.04 using Nvidia CUDA drivers

Fresh install of 22.04 LTS here, using 4090 RTX as main card. I manually installed the CUDA Nvidia drivers (running some machine learning code locally that relies on it). Either I installed 535.04 Nvidia driver alongside this, or they're packaged together. Either way, my system is reporting that it's using 535.04 from what I can see.

Everything seems to be working correctly regarding CUDA, but when I move my windows around inside Ubuntu it's very laggy, despite being on a 120hz monitor it's super choppy (mouse movement is smooth, window movement is stuttering/choppy). For example, moving the terminal window around.

Do I also need to install some additional Nvidia display drivers alongside whatever drivers I'm using right now, or is there another cause?

(base) me@me:~$ dpkg -l | grep -i nvidia
ii  cuda-nsight-compute-12-2                   12.2.0-1                                amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-12-2                   12.2.0-1                                amd64        NVIDIA Nsight Systems
ii  cuda-nvtx-12-2                             12.2.53-1                               amd64        NVIDIA Tools Extension
ii  libnvidia-cfg1-535:amd64                   535.54.03-0ubuntu1                      amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-535                       535.54.03-0ubuntu1                      all          Shared files used by the NVIDIA libraries
ii  libnvidia-compute-535:amd64                535.54.03-0ubuntu1                      amd64        NVIDIA libcompute package
ii  libnvidia-compute-535:i386                 535.54.03-0ubuntu1                      i386         NVIDIA libcompute package
ii  libnvidia-decode-535:amd64                 535.54.03-0ubuntu1                      amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-decode-535:i386                  535.54.03-0ubuntu1                      i386         NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-535:amd64                 535.54.03-0ubuntu1                      amd64        NVENC Video Encoding runtime library
ii  libnvidia-encode-535:i386                  535.54.03-0ubuntu1                      i386         NVENC Video Encoding runtime library
ii  libnvidia-extra-535:amd64                  535.54.03-0ubuntu1                      amd64        Extra libraries for the NVIDIA driver
ii  libnvidia-fbc1-535:amd64                   535.54.03-0ubuntu1                      amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-fbc1-535:i386                    535.54.03-0ubuntu1                      i386         NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-535:amd64                     535.54.03-0ubuntu1                      amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-gl-535:i386                      535.54.03-0ubuntu1                      i386         NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  nsight-compute-2023.2.0                    2023.2.0.16-1                           amd64        NVIDIA Nsight Compute
ii  nvidia-compute-utils-535                   535.54.03-0ubuntu1                      amd64        NVIDIA compute utilities
ii  nvidia-dkms-535                            535.54.03-0ubuntu1                      amd64        NVIDIA DKMS package
ii  nvidia-driver-535                          535.54.03-0ubuntu1                      amd64        NVIDIA driver metapackage
ii  nvidia-fs                                  2.16.1-1                                amd64        NVIDIA filesystem for GPUDirect Storage
ii  nvidia-fs-dkms                             2.16.1-1                                amd64        NVIDIA filesystem DKMS package
ii  nvidia-gds                                 12.2.0-1                                amd64        GPU Direct Storage meta-package
ii  nvidia-gds-12-2                            12.2.0-1                                amd64        GPU Direct Storage 12.2 meta-package
ii  nvidia-kernel-common-535                   535.54.03-0ubuntu1                      amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-535                   535.54.03-0ubuntu1                      amd64        NVIDIA kernel source package
ii  nvidia-modprobe                            535.54.03-0ubuntu1                      amd64        Load the NVIDIA kernel driver and create device files
ii  nvidia-prime                               0.8.17.1                                all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                            535.54.03-0ubuntu1                      amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-535                           535.54.03-0ubuntu1                      amd64        NVIDIA driver support binaries
ii  screen-resolution-extra                    0.18.2                                  all          Extension for the nvidia-settings control panel
ii  xserver-xorg-video-nvidia-535              535.54.03-0ubuntu1                      amd64        NVIDIA binary Xorg driver

(base) me@me:~$ lspci -nnk | grep -iA2 vga
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2684] (rev a1)
    Subsystem: ASUSTeK Computer Inc. Device [1043:889c]
    Kernel driver in use: nvidia
--
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2203] (rev a1)
    Subsystem: ASUSTeK Computer Inc. Device [1043:8875]
    Kernel driver in use: nvidia

(base) me@me:~$ sudo lshw -class video | grep driver=
       configuration: driver=nvidia latency=0
       configuration: driver=nvidia latency=0
(base) me@me:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  535.54.03  Tue Jun  6 22:20:39 UTC 2023
GCC version:  gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)