SSH Bind Address - Cannot Assign Requested Address

I've got 2 NIC's, and just want to SSH from one specific interface. As you can see, the -b option in ssh doesn't seem to be able to use my address. Not sure why?

$ ip a show enp8s0 | grep "default\|inet "
2: enp8s0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master INT-VRF1 state UP group default qlen 1000
    inet 172.31.16.22/24 brd 172.31.16.255 scope global dynamic noprefixroute enp8s0

$ ping -I enp8s0 172.31.16.10
PING 172.31.16.10 (172.31.16.10) from 172.31.16.22 enp8s0: 56(84) bytes of data.
64 bytes from 172.31.16.10: icmp_seq=1 ttl=255 time=0.627 ms
64 bytes from 172.31.16.10: icmp_seq=2 ttl=255 time=2.49 ms

$ ssh -b 172.31.16.22 172.31.16.10
bind 172.31.16.22: Cannot assign requested address
ssh: connect to host 172.31.16.10 port 22: failure

It's like SSH is unaware of the VRF. I thought this could be a namespace issue so tried also using the kernel settings below to no avail.

$ sudo sysctl -a | grep l3mdev
net.ipv4.raw_l3mdev_accept = 1
net.ipv4.tcp_l3mdev_accept = 1
net.ipv4.udp_l3mdev_accept = 1

Added below as per request of comment:

$ ip -4 addr list | grep 172.31.16.22
inet 172.31.16.22/24 brd 172.31.16.255 scope global dynamic noprefixroute enp8s0

UPDATE

I did manage to get this working via the below:

$ sudo ip vrf exec INT-VRF1 ssh 172.31.16.10

However, I can only get it working with sudo. Otherwise I get:

$ ip vrf exec INT-VRF1 ssh 172.31.16.10
Failed to load BPF prog: 'Operation not permitted'

Not sure why really. But I just need to be able to use client apps like SSH over this interface. How/What is the way to do it?

The only thing I can seem to find online is about binding the application's SO_BINDTODEVICE to a certain interface in order to force a socket to be created on a specific NIC. But I'm really not sure how to go about setting this up with for an SSH client connection.