Not too long ago I put together a 40G network using a Mikrotik CRS354-48G-4S+2Q+RM switch as the backbone of it. I very quickly ran into an issue using this switch, transferring data from a port with more bandwidth than it's destination port causes a very large drop in throughput. For example, my personal Windows machine is equipped with 10G; I have full gigabit internet connected via 1G. Running a speed test will give me expected ~900Mbps down but only ~350Mbps up. Stepping my 10G card & switch port down to 1G - it works at full speed. Similarly, transferring 40G to a 10G port results in a significant reduction of throughput.
My solution since I've put the network together has been to setup routing for each connected device using either 40G or 10G. In this routing, the device uses it's 1G onboard for internet bound traffic and 40G/10G for local traffic. Since this is where the majority of the instances occur. However, this has not been very reliable.
I need a solution that is contained within the switch, preferably. This particular switch is able to run RouterOS or SwitchOS (I've updated and tested both before posting here). I've been told this is likely not possible due to a hardware limitation. In which case I need a more reliable routing solution.
I've been using net-tools' route command to adjust the routing rules. My route -n looks like this;
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 1 0 0 vmbr0
0.0.0.0 192.168.0.1 0.0.0.0 UG 1000 0 0 vmbr1
192.168.0.0 0.0.0.0 255.255.255.0 U 1 0 0 vmbr1
192.168.0.0 0.0.0.0 255.255.255.0 U 1000 0 0 vmbr0
Where vmbr0 = 1G and vmbr1 = 40G
This workaround only tends to work for one local device at a time. This wasn't a huge deal until recently when I put in a second server. The two servers connected via 40G will have no issue with the routing, even when the 1G NIC IP is used. However, when I connect my 10G client, it will route to the 1G regardless of the IP or what interface/IP the server program is bound to. If I restart networking within one of the 40G servers, I'm temporarily able to send full 10G from my client. This is sadly not consistent and will switch back to using the 1G at any given moment.
If I'm using routing properly and it continues to be stubborn, some sort of network limiter based on IPv4 addresses may be ideal.
Details of the devices on my network can be found here.
Any help in this matter will be greatly appreciated.
