Unable to install and use NVIDIA drivers on a Ubuntu virtual machine

I have a Dell tower with two RTX 3060s, and a few Ubuntu 20.04 VMs set up from this machine. Two VMs each have an RTX 3060 assigned to them, with GPU passthrough. However, regardless of how many times I try to uninstall and re-install NVIDIA drivers, I get the following error:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

I have another computer, also with an NVIDIA GPU, where I was able to quickly install NVIDIA drivers with no problems. This other computer is just a normal computer, not a VM with a passthrough'd GPU. Are there any steps I am missing, that are specific to installing NVIDIA drivers on a Ubuntu virtual machine?

Not sure if this information is useful, but lspci -nnk yields the following:

00:00.0 Host bridge [0600]: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge [8086:7190] (rev 01)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
        Kernel driver in use: agpgart-intel
00:01.0 PCI bridge [0604]: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge [8086:7191] (rev 01)
00:07.0 ISA bridge [0601]: Intel Corporation 82371AB/EB/MB PIIX4 ISA [8086:7110] (rev 08)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
00:07.1 IDE interface [0101]: Intel Corporation 82371AB/EB/MB PIIX4 IDE [8086:7111] (rev 01)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
        Kernel driver in use: ata_piix
        Kernel modules: pata_acpi
00:07.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 08)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
        Kernel modules: i2c_piix4
00:07.7 System peripheral [0880]: VMware Virtual Machine Communication Interface [15ad:0740] (rev 10)
        Subsystem: VMware Virtual Machine Communication Interface [15ad:0740]
        Kernel driver in use: vmw_vmci
        Kernel modules: vmw_vmci
00:0f.0 VGA compatible controller [0300]: VMware SVGA II Adapter [15ad:0405]
        Subsystem: VMware SVGA II Adapter [15ad:0405]
        Kernel driver in use: vmwgfx
        Kernel modules: vmwgfx
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2504] (rev a1)
        DeviceName: pciPassthru0
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:397d]
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:228e] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:397d]
        Kernel modules: snd_hda_intel

I used ESXi to spawn the virtual machines, and I am willing to answer any questions about my set up.

Ve a caracteristicas de sistema y mira que los hipervisores estèn todos marcados, hay varias opciones. A mi me pasò lo mismo, aun asi las maquinas virtuales no se pueden comparar a la instacion dual o local, la experiencia es muy diferente, pruebalo en dual o en un usb de 62gb y dale el maximo de persistencia, es lo mas parecido a instalarlo, lo prubas y seguro que te gusta, saludos