Can't get nouveau to work with VA_API for video acceleration on firefox? Only can get CPU drivers to work with firefox

Edit2: Sorry, running Ubuntu 20.04.3 LTS, and HWE stack. I get the same error with the GA Stack.

_

I've been trying to get nouveau to work so that I can use my Nvidia P500 for hardware acceleration instead of my CPU.

This is the error I'm getting:

(base) wallheck@wallheck:~$ export LIBVA_DRIVER_NAME=nouveau
(base) wallheck@wallheck:~$ vainfo
libva info: VA-API version 1.7.0
libva info: User environment variable requested driver 'nouveau'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nouveau_drv_video.so
libva info: Found init function __vaDriverInit_1_7
iris: driver missing
iris: driver missing
libva error: /usr/lib/x86_64-linux-gnu/dri/nouveau_drv_video.so init failed
libva info: va_openDriver() returns 2
vaInitialize failed with error code 2 (resource allocation failed),exit

I installed nouveau with

sudo apt install nouveau-firmware
sudo apt install xserver-xorg-video-nouveau

and for both it says its already the newest version.

I am able to get video acceleration to work with the CPU using this guide: https://ubuntuhandbook.org/index.php/2021/08/enable-hardware-video-acceleration-va-api-for-firefox-in-ubuntu-20-04-18-04-higher/. Using i965 drivers, I get activity under the video row when I use sudo intel_gpu_top.

However, when I use nvtop, I cannot get any activity for my GPU.

Here's the kernel drivers for the integrated graphics and GPU:

(base) wallheck@wallheck:~$ lspci -k | grep -EA3 'VGA|3D|Display'
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 620 (rev 07)
    Subsystem: Lenovo UHD Graphics 620
    Kernel driver in use: i915
    Kernel modules: i915
--
02:00.0 3D controller: NVIDIA Corporation GP108GLM [Quadro P500 Mobile] (rev a1)
    Subsystem: Lenovo GP108GLM [Quadro P500 Mobile]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

I don't think I need to change the kernel driver to nouveau as the intel drivers worked with i965 specified and i915 kernel in use.

Not really sure what to do from here.

Okay, the following is probably not the desired solution by the OP. But I would look at the whole thing from a somewhat different perspective.

At first, so far I know the VA-API support is (still) not fully mature in the nouveau driver. Only VDPAU is avaiable as the reliable solution but that one cannot be used in Firefox (and under Wayland).

However, why must the video decoding be processed via the Nvidia GPU when an Intel iGPU which includes the Intel Quick Sync Video functionality is present? I think that should be here the way to go. In my case DRI_PRIME=0 firefox starts firefox at GPU 0 while DRI_PRIME=1 firefox will start it at GPU 1.

The problem is that this principle doesn't work with the classic i965 driver, it only works with "normal" Gallium3D based drivers. Here will be the new crocus driver an option, it is for sure compatible with nouveau (which is itself Gallium based). Unfortunately crocus lacks at the moment any VA-API support, that one should be added during the next months. Note, that new driver can be tested via the Oibaf PPA:

sudo add-apt-repository ppa:oibaf/graphics-drivers
sudo apt-get update

Finally, if any attempt to use the Intel Quick Sync Video feature via the i965 driver is failing I see only one last workaround. The NVIDIA GPU has to be disabled temporary in the Lenovo firmware. This will then effectively disable the nouveau driver and everything will fall back to the i965 classic one. I know this sucks but this will allow VA-API video decoding via Intel Quick Sync Video in Firefox. And when crocus is finished, the switching between the GPU and their capabilities should be no longer a problem. ;-)