How to make secondary IP routable

I am trying to assign a secondary IP to my primary interface.

I added the IP to my eth0 interface

ip addr add 172.20.14.253/21 dev eth0

$ ip a show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether fa:16:3e:35:91:b5 brd ff:ff:ff:ff:ff:ff
    inet 172.20.11.35/21 brd 172.20.15.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.20.11.253/21 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe35:91b5/64 scope link
       valid_lft forever preferred_lft forever

now if I try to ping this IP from my local box I see that the server is getting ARP requests and it is responding too, but the response is not getting through.

$ tcpdump -A -nnvvv host 172.20.11.253
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:50:42.746806 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.11.253 tell 172.20.10.198, length 46
..........>.^X..
.............................
09:50:42.746864 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.11.253 is-at fa:16:3e:35:91:b5, length 28
..........>5........>.^X..
.
09:50:43.777926 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.11.253 tell 172.20.10.198, length 46
..........>.^X..
.............................
09:50:43.777956 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.11.253 is-at fa:16:3e:35:91:b5, length 28
..........>5........>.^X..
.
09:50:44.805936 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.11.253 tell 172.20.10.198, length 46
..........>.^X..
.............................
09:50:44.805994 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.11.253 is-at fa:16:3e:35:91:b5, length 28
..........>5........>.^X..

I assumed there is no route on the remote box to respond for the requests that land on 172.20.11.253 so I added a route as well.

ip route add 172.20.10.198/32 via default dev eth0 src 172.20.11.253

but nothing changed.

my route

$ ip r
default via 172.20.15.253 dev eth0
10.0.0.0/8 via 172.20.8.1 dev eth0
172.16.0.0/12 via 172.20.8.1 dev eth0
172.20.8.0/21 dev eth0 proto kernel scope link src 172.20.11.35
172.20.10.198 dev eth0 src 172.20.11.253
192.168.0.0/20 dev docker0 proto kernel scope link src 192.168.0.1 linkdown

What am I missing here?