The internal IP address of my router changed, and ever since TrueNAS has been misbehaving. None of the jails can resolve a hostname anymore. In troubleshooting this, I discovered that the client cannot ping any external IP address without the response being from the host.
For info:
- In this example, the client jail's IP is 10.153.2.250
- The host's (FreeNAS) IP is 10.153.2.9
- The new gateway is 10.0.55.1 (old was 10.55.0.1), and this has been updated in both the host's network configuration and the jails'.
- When the gateway IP changed, FreeNAS got into a boot loop on the next startup ... apparently to a DNS resolution failure. As I recall (it was 3am and my mind was foggy), updating the nameserver and route finally fixed it.
- An SMB share on the host stopped working at the same time (and still hasn't come back)
Cannot ping domain names:
round-trip min/avg/max/stddev = 15.553/15.790/16.027/0.237 ms
root@jail:~ # ping google.com
ping: cannot resolve google.com: Host name lookup failure
If manually I tell the jail client to look up a domain, I get an IP address back:
root@jail:~ # host google.com
google.com has address 172.217.15.110
google.com has IPv6 address 2607:f8b0:4009:801::200e
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 8 smtp.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
HOWEVER, for any IP address that I ping from within the jail client, the response is from the IP of the jail's host (FreeNAS). Pinging the same IPs from the host gets a normal response.
root@jail:~ # ping 172.217.15.110
PING 172.217.15.110 (172.217.15.110): 56 data bytes
64 bytes from 10.153.2.9: icmp_seq=0 ttl=116 time=30.454 ms
64 bytes from 10.153.2.9: icmp_seq=1 ttl=116 time=30.235 ms
64 bytes from 10.153.2.9: icmp_seq=2 ttl=116 time=30.534 ms
^C
--- 172.217.15.110 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 30.235/30.408/30.534/0.126 ms
Another example (note the host's IP, 10.153.2.9, in the response):
root@jail:~ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 10.153.2.9: icmp_seq=0 ttl=57 time=16.027 ms
64 bytes from 10.153.2.9: icmp_seq=1 ttl=57 time=15.553 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
But, if I ping another internal IP (other than that of the router), I get the expected response:
root@jail:~ # ping 10.153.2.205
PING 10.153.2.205 (10.153.2.205): 56 data bytes
64 bytes from 10.153.2.205: icmp_seq=0 ttl=64 time=0.295 ms
64 bytes from 10.153.2.205: icmp_seq=1 ttl=64 time=0.237 ms
64 bytes from 10.153.2.205: icmp_seq=2 ttl=64 time=0.193 ms
^C
--- 10.153.2.205 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.193/0.242/0.295/0.042 ms
root@jail:~
Here is the result of netstat -rn from within the host. I'm not too familiar with this output, but nothing sticks out to me as amiss (and it closely resembles the host's version):
root@jail:~ # netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.0.55.1 UGS epair0b
10.0.0.0/8 link#3 U epair0b
10.153.2.250 link#3 UHS lo0
127.0.0.1 link#1 UH lo0
Internet6:
Destination Gateway Flags NetifExpire
::/96 ::1 UGRS lo0
::1 link#1 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#1 U lo0
fe80::1%lo0 link#1 UHS lo0
ff02::/16 ::1 UGRS lo0
Here are the contents of the client's resolv.conf (identical to the host's resolv.conf, and predictably matches the settings in the FreeNAS network configuration):
root@jail:~ # cat /etc/resolv.conf
# Generated by resolvconf
search local
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 10.0.55.1
root@jail:~
I checked the arp cache and the MAC addresses are correct.
SO....why would the jail be getting the host's IP back in a ping response?
