ARP Weirdness - Missing ARP replies?

Ex Umbris

8/1/23, 8:54 PM

Here's a picture of the relevant portion of my network.

The problem is as follows:

The Windows PC at left and the Synology NAS are both able to reach the two webcams (ping) and the ARP tables on both devices are correct.
On both the Linux Server and Linux PC at the bottom right, pinging either webcam results in "No route to host".
The ARP tables on the Linux Server and Linux PC do not have entries for the webcam addresses.
Monitoring ARP traffic with tcpdump, I see the two Linux machines repeatedly sending out ARP requests for the webcam IPs, but no response is received.
As a workaround I have found that I can manually add the required ARP entries on the Linux systems, which allows them to communicate with the cameras. This tells me it's purely an ARP issue as the connectivity works if the ARP entries are there.

The Linux machines are not running iptables or nftables firewalls. There's only one subnet and one L2 net.

My guess at this point is that the Netgear GS308 switch is defective, but I struggle to come up with a failure mode that would prevent ARP replies but still allow communication. The GS308 is unmanaged, so it has no configuration options.

When I'm next at the site I will replace the switch to see if it's the culprit.

Any suggestions as to what might cause this, and/or possible further troubleshooting steps?

UPDATE: I can no longer troubleshoot this because the problem "fixed itself" after about 24 hours. I hate it when things break and then fix themselves without me understanding what happened :-(

I am voting to close the question as unreproducible.

0 + 0

arp

Ex Umbris

8/1/23, 9:13 PM

To the close voter: I've already tried NetworkEngineering. They said it's off-topic there. This is a business network environment (I simplified the network diagram to show only the relevant devices). If it's off-topic here and on NetworkEngineering, then where is it on-topic.

Mark Wagner

8/1/23, 10:50 PM

Looking around a bit, the zyxel c3000z device appears to be accessible via ssh and runs linux. You may be able to log into it and use typical linux commands to troubleshoot (though I've also seen a claim that Century Link has locked this down.)

Ex Umbris

8/2/23, 12:11 AM

My first thought was that the router was the issue so logged in to the router via SSH, it's not locked down at all. It has arp entries for the webcams. If the router were the problem I would expect the issue to occur for all hosts on either GS308. But the problem occurs only on the Linux hosts on one GS308, but not on the NAS (which also runs Linux) on the same switch. And to top it off, adding the missing arp entries via `sudo arp -a [ip] [mac]` "cures" the problem.

Hagen von Eitzen

8/2/23, 6:33 AM

Can you follow the arp query broadcasts further by sniffing at other segments? Such as: Do the arp queries get past the first switch? Do they get past the second switch? Would a different device in place of the webcams be reachable? Can you trace the arp replies back at every switch (requires you use a monitor port at the switch as the replies are unicast)?

Appleoddity

8/2/23, 5:13 PM

What exactly do you see in the ARP request? Please post the contents of the ARP request. Have you double checked the subnet mask on the webcams? Or perhaps a wireless policy that results in network isolation?

Ex Umbris

8/2/23, 9:07 PM