Please explain this traceroute

What might cause the middle two RTT to timeout. This happens every time. Users are reporting slow network access from the remote site. We have contacted Meraki and they have not found anything wrong. We have replaced cables. We have contacted the ISP of the ELAN (nothing wrong). We have rebooted the stacks. These traceroutes have been performed with no users on the network. Normally there are about 15 users in the office.

Host1 Windows 10

Host2 Server 2019

Layer 3 switches Meraki MS250-48FP Current firmware

Host1 IP is 10.10.1.80 plugged directly to switch2

Host2 IP is 10.0.0.20 plugged directly to switch1

ELAN connection 172.16.255.2 plugged directly to switch2 (10.10.1.1)

ELAN connection 172.16.255.1 plugged directly to switch1 (10.0.0.1)

10.10.1.1 switch2 Route 10.0.0.0/24 next hop 172.16.255.1

10.0.0.1 switch1 Route 10.10.0.0/16 next hop 172.16.255.2

Host1 (10.10.1.80) <--Cat5--> (10.10.1.1) Switch2 (172.16.255.2) <--ELAN--> (172.16.255.1) Switch1 (10.0.0.1) <--Cat5--> Host2 (10.0.0.20)

From 10.10.1.80

tracert 10.0.0.20

Tracing route to 10.0.0.20 over a maximum of 30 hops:

  1     1 ms     *        1 ms  10.10.1.1
  2    26 ms     *       26 ms  172.16.255.1
  3    27 ms    26 ms    26 ms  10.0.0.20

Trace complete.

From 10.0.0.20

tracert 10.10.1.80

Tracing route to 10.10.1.80 over a maximum of 30 hops

  1    <1 ms     *       <1 ms  10.0.0.1
  2    27 ms     *       27 ms  172.16.255.2
  3    27 ms    27 ms    27 ms  10.10.1.80

Trace complete.

What might cause the middle two RTT to timeout.

The most likely causes include:

• those hops are limiting ICMP Time exceeded messages
• packets are lost on the way to the hop (unlikely when later hops are fine)
• ICMP messages are lost on the way back (rather unlikely but possible when there's ICMP quenching anywhere)

Simply put, traceroute isn't really a good tool to diagnose packet loss.

Better alternatives are:

• Check intermediate switches for frame loss due to link issues (receive failure in interface statistics), congestion (frame drops in interface statistics), or STP reconvergence (event log). It's generally a good idea to trigger SNMP traps when significant frame drops occur anywhere.
• Check intermediate routers for packet loss (similar to above).
• Check overall traffic and individual flows using Netflow or sFlow.

Users are reporting slow network access from the remote site.

When there's another network (that you don't control) in between, things are a bit harder. You can try to use pathping or traceroute outside the VPN tunnel (between ISP handovers), or you could run a diagnostic stream along with your tunnel (it doesn't need to use high bandwidth usually). UDP would be a good choice as it doesn't hide any losses by retransmitting. Testing through the tunnel hides a lot of information as the tunnel is just a single hop.

If both sites use the same ISP you can collect some data and contact support. If you use different ISPs it'll be hard to point fingers.