USB 3 stick detected on 480M bus when plugged in after boot, but on 5000M bus before boot

I have a strange problem on my laptop (Gigabyte Aero 15 Classic XA) running Ubuntu 21.10 (was the same on 20.04). When I plug in a USB 3 stick I get only USB 2 speeds. But when the same stick was inserted prior to boot, I get USB 3 speed.

lsusb -t confirms:

Stick connected after boot:

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 10000M
|__ Port 5: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 3: Dev 14, If 0, Class=Mass Storage, Driver=usb-storage, 480M <- the stick
|__ Port 9: Dev 9, If 0, Class=Video, Driver=uvcvideo, 480M
|__ Port 9: Dev 9, If 1, Class=Video, Driver=uvcvideo, 480M
|__ Port 11: Dev 10, If 2, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 11: Dev 10, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 11: Dev 10, If 3, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 11: Dev 10, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 14: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 14: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M

And prior to boot:

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 10000M
|__ Port 3: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 5: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M <- the stick
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 9: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
|__ Port 9: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
|__ Port 11: Dev 3, If 2, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 11: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 11: Dev 3, If 3, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 11: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 14: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 14: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M

Any USB stick is affected. So what is causing this? How can the same stick on the same physical USB port be on different busses, depending on when it was inserted?

System Information

  PROCESSOR:              Intel Core i7-9750H @ 4.50GHz
    Core Count:           6                                                 
    Thread Count:         12                                                
    Extensions:           SSE 4.2 + AVX2 + AVX + RDRAND + FSGSBASE          
    Cache Size:           12288 MB                                          
    Microcode:            0xea                                              
    Core Family:          Kaby/Coffee/Whiskey Lake                          
    Scaling Driver:       intel_pstate powersave (EPP: balance_performance) 

  GRAPHICS:               Gigabyte Intel UHD 630 CFL GT2 8GB
    BAR1 / Visible vRAM:  256 MiB          
    OpenGL:               4.6 Mesa 21.2.6  
    Display Driver:       NVIDIA           
    Monitor:              LQ156M1JW03      
    Screen:               1920x1080        

  MOTHERBOARD:            GIGABYTE AERO 15 Classic-XA
    BIOS Version:         B04                                                             
    Chipset:              Intel Cannon Lake PCH                                           
    Audio:                Realtek ALC255                                                  
    Network:              Qualcomm Atheros Killer E2500 + Intel Cannon Lake PCH CNVi WiFi 

Greetings, Simon