I have a similar problem (Ubuntu 18), and this is how I solved it.
The pulse audio system has a module that automatically switches audio devices when a new device is plugged in. Typically, this is helpful, especially when you plug in headphones and unplug them. I found disabling this module is not really helpful. Newer versions of pulseaudio support blacklisting devices, but I think the following is sometimes a better solution and works with older versions of pulseaudio.
In a few cases, devices that are not primarily audio (such as your usb hub, and video cards with HDMI audio capability) make this messy. The system has priories to fix this, but USB audio devices have the highest priority.
Fixing this involves creating two files for your device. For explanation purposes, let's call this device 'myunusedaudio'.
The first file is /etc/udev/rules.d/99-myunusedaudio.rules
This file needs to include a udev rule to recognize the device.
Typically, you would want to match it with one or more attributes such as
ATTRS{idProduct} ATTRS{idVendor} ATTRS{serial}
The match line should include the attribute ENV{PULSE_PROFILE_SET}="myunusedaudio.conf" which links it to the next file. Please check with other guides on how to build and test a udev match rule for your device. If you make changes to this file, generally unplugging the usb device and plugging it back in will make the changes take effect.
The second file would be:
/usr/share/pulseaudio/alsa-mixer/profile-sets/myunusedaudio.conf (Note filename match with above.)
In this file, you set a priority so that your device is the lowest priority and will never be a default device. The file contents should be:
[General]
priority = 1
After these two files are in place, your audio device should no longer be chosen as the default when you plug it in.
