Ubuntu server 20.04 with netplan and two 1G ports that are on the same subnet

I am getting collocation service for my hardware that has 4x1G ports on the same network card. Two of the ports are plugged and they correspond to eno5 and eno6 interfaces on my operating system which is Ubuntu server 20.04. The provider has assigned me a /24 ipv4 subnet and they say I can use both ports on it. Lets consider xx.xx.xx.0/24 is my ip range and xx.xx.xx.1 is my gateway. The scenario I want to run is to assign xx.xx.xx.2 to eno5 and xx.xx.xx.3 to eno6. The behavior I expect is that the traffic coming to xx.xx.xx.3 from eno6 also go out through eno6.

I have come up with following netplan config so far which is adjusted within two seperate config files:

Config related to eno5 is /etc/netplan/conf5.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno5:
      dhcp4: false
      dhcp6: false
      addresses:
      - xx.xx.xx.2/24
      nameservers:
        addresses:
        - 8.8.8.8
        - 8.8.4.4
      routes:
        - to: 0.0.0.0/0
          via: xx.xx.xx.1
          table: 500
          metric: 500
      routing-policy:
        - from: xx.xx.xx.2/24
          table: 500

Config related to eno6 is /etc/netplan/conf6.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno6:
      dhcp4: false
      dhcp6: false
      addresses:
      - xx.xx.xx.3/24
      nameservers:
        addresses:
        - 8.8.8.8
        - 8.8.4.4
      routes:
        - to: 0.0.0.0/0
          via: xx.xx.xx.1
          table: 600
          metric: 600
      routing-policy:
        - from: xx.xx.xx.3/24
          table: 600

Now, I have two IPs working. I mean if I ping from a remote machine both xx.xx.xx.2 and xx.xx.xx.3 respond successfully. I also can run successful pings inside my machine:

ping -I eno5 google.com
ping -I eno6 google.com

Every thing is good up until I want to ssh to xx.xx.xx.3 from a remote machine and download a large file. When I try to do so after that the download is complete I don't see eno6 tx count moved much. Instead tx count on eno5 moves hugely. I check the counts using:

> ip -s link show eno5
link/ether d4:f5:ef:4b:54:6c brd ff:ff:ff:ff:ff:ff
RX: bytes  packets  errors  dropped overrun mcast   
237309112    35714    0       901     0       913     
TX: bytes  packets  errors  dropped carrier collsns 
991989133    54400     0       0       0       0  
> ip -s link show eno6
link/ether d4:f5:ef:4b:54:6d brd ff:ff:ff:ff:ff:ff
RX: bytes  packets  errors  dropped overrun mcast   
153728525    19980    0       933     0       945     
TX: bytes  packets  errors  dropped carrier collsns 
2582       29       0       0       0       0  

I conclude that the ssh request I am sending to xx.xx.xx.3 (eno6) is responded through eno5. Am I right?

Which part of my policy-routing and or the whole config is wrong that I am ending up like this?

Please help.