Destination Host Unreachable: Failover IP Addresses

I have added two additional IP addresses following this guide: https://docs.ovh.com/ie/en/dedicated/network-ipaliasing/#debian-9-ubuntu-1704-and-arch-linux_1

cat /etc/systemd/network/50-default.network

[Match]
MACAddress=08:(...)

[Network]
Description=network interface on public network, with default route
DHCP=no
Address=37.187.90.XX/24
Gateway=37.187.90.254
IPv6AcceptRA=no
NTP=ntp.ovh.net
DNS=127.0.0.1
DNS=213.186.33.99
DNS=2001:41(...)
Gateway=2001:41(...)

[Address]
Address=2001:41d(...)

[Address]
Address=188.165.20.XXX
Label=failover1

[Address]
Address=94.23.157.XXX
Label=failover2

[Route]
Destination=2001:41(...)
Scope=link

Everything it's working fine but I can't ping 94.152.8.40 (external server from different ISP)

ping -c 4 94.152.8.40

PING 94.152.8.40 (94.152.8.40) 56(84) bytes of data.
From 94.23.157.XXX icmp_seq=1 Destination Host Unreachable
From 94.23.157.XXX icmp_seq=2 Destination Host Unreachable
From 94.23.157.XXX icmp_seq=3 Destination Host Unreachable
From 94.23.157.XXX icmp_seq=4 Destination Host Unreachable
(...)

I can ping 94.152.8.40 using -I:

ping -c4 -I 37.187.90.XX 94.152.8.40

PING 94.152.8.40 (94.152.8.40) from 37.187.90.XXX : 56(84) bytes of data.
64 bytes from 94.152.8.40: icmp_seq=1 ttl=52 time=32.6 ms
64 bytes from 94.152.8.40: icmp_seq=2 ttl=52 time=32.0 ms
64 bytes from 94.152.8.40: icmp_seq=3 ttl=52 time=32.0 ms
64 bytes from 94.152.8.40: icmp_seq=4 ttl=52 time=32.0 ms
(...)

ip route show to match 94.152.8.40

default via 37.187.90.254 dev enp3s0 proto static 
94.0.0.0/8 dev enp3s0 proto kernel scope link src 94.23.157.XXX

netstat -nr

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         37.187.90.254   0.0.0.0         UG        0 0          0 enp3s0
37.187.90.0     0.0.0.0         255.255.255.0   U         0 0          0 enp3s0
94.0.0.0        0.0.0.0         255.0.0.0       U         0 0          0 enp3s0
188.165.0.0     0.0.0.0         255.255.0.0     U         0 0          0 enp3s0

ip route list

default via 37.187.90.254 dev enp3s0 proto static 
37.187.90.0/24 dev enp3s0 proto kernel scope link src 37.187.90.XX 
94.0.0.0/8 dev enp3s0 proto kernel scope link src 94.23.157.XXX 
188.165.0.0/16 dev enp3s0 proto kernel scope link src 188.165.20.XXX 

When I disable 94.23.157.XXX in /etc/systemd/network/50-default.network

   #[Address]
   #Address=94.23.157.XXX
   #Label=failover2

and restart systemctl restart systemd-networkd , I can now ping 94.152.8.40 without any problems.

ping -c 4 94.152.8.40

PING 94.152.8.40 (94.152.8.40) 56(84) bytes of data.
64 bytes from 94.152.8.40: icmp_seq=1 ttl=52 time=32.0 ms
64 bytes from 94.152.8.40: icmp_seq=2 ttl=52 time=32.0 ms
64 bytes from 94.152.8.40: icmp_seq=3 ttl=52 time=32.0 ms
64 bytes from 94.152.8.40: icmp_seq=4 ttl=52 time=32.1 ms

netstat -nr

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         37.187.90.254   0.0.0.0         UG        0 0          0 enp3s0
37.187.90.0     0.0.0.0         255.255.255.0   U         0 0          0 enp3s0

ip route list

default via 37.187.90.254 dev enp3s0 proto static 
37.187.90.0/24 dev enp3s0 proto kernel scope link src 37.187.90.XX 

ip route show to match 94.152.8.40

default via 37.187.90.254 dev enp3s0 proto static 

How to disable both routes

94.0.0.0/8 dev enp3s0 proto kernel scope link src 94.23.157.XXX 
188.165.0.0/16 dev enp3s0 proto kernel scope link src 188.165.20.XXX 

without disabling additional IP (FailOver) and just use one default 37.187.90.0/24 gateway for all destinations

Thanks, Dawid