Symptom: The first attempt at connecting to a TCP service (https, ftp, or ssh) fails. The second attempt succeeds Subsequent connections will work until there is a 15 minute period of inactivity, in which case the next connection will fail again. Upon inspection of packets on both sides, the client seems to log a TCP 3 way handshake for the first connection followed by a reset. The server logs no packets. Strangely, this behavior is the same even if the client connects to a port the server is not listening on (for this example I used port 234). Subsequent connections to the non-existent service on port 234 are logged as SYN/Resets, and traffic to supported ports is logged as expected.
Command used to connect to port 234:
$ nc server-ip 234
executed three times
Server is running Ubuntu 20.04.6 LTS and hosted by GoDaddy
Packet dump on client:
1. 11:00:26.963058 IP client-ip.65228 > server-ip.234: Flags [S], seq 4286801981, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 866207816 ecr 0,sackOK,eol], length 0
2. 11:00:27.012703 IP server-ip.234 > client-ip.65228: Flags [S.], seq 1144885311, ack 4286801982, win 16384, options [mss 1460], length 0
3. 11:00:27.012755 IP client-ip.65228 > server-ip.234: Flags [.], ack 1, win 65535, length 0
4. 11:00:27.055571 IP server-ip.234 > client-ip.65228: Flags [R.], seq 1, ack 1, win 16384, length 0
5. 11:00:29.631388 IP client-ip.65229 > server-ip.234: Flags [S], seq 255589775, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 145472704 ecr 0,sackOK,eol], length 0
6. 11:00:29.675475 IP server-ip.234 > client-ip.65229: Flags [R.], seq 0, ack 255589776, win 0, length 0
7. 11:00:32.484897 IP client-ip.65230 >server-ip.234: Flags [S], seq 1572449604, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1092910203 ecr 0,sackOK,eol], length 0
8. 11:00:32.528987 IP server-ip.234 > client-ip.65230: Flags [R.], seq 0, ack 1572449605, win 0, length 0
Packet dump on server (note a 3 hour time difference):
1. no corresponding packets
2. no corresponding packets
3. no corresponding packets
4. no corresponding packets
5. 08:00:29.712757 IP client-ip.65229 > server-ip.234: Flags [S], seq 255589775, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 145472704 ecr 0,sackOK,eol], length 0
6. 08:00:29.712785 IP server-ip.234 > client-ip.65229: Flags [R.], seq 0, ack 255589776, win 0, length 0
7. 08:00:32.566058 IP client-ip.65230 > server-ip.234: Flags [S], seq 1572449604, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1092910203 ecr 0,sackOK,eol], length 0
8. 08:00:32.566085 IP server-ip.234 > client-ip.65230: Flags [R.], seq 0, ack 1572449605, win 0, length 0
Any ideas?
The local firewall is configured like so:
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
Here is a partial traceroute to the server:
4 be-50-rar01.kokomo.in.indiana.comcast.net (96.108.120.145) 18.098 ms 18.663 ms 21.537 ms
5 24.153.88.85 (24.153.88.85) 26.530 ms 23.513 ms 23.843 ms
6 * 4.68.110.122 (4.68.110.122) 34.668 ms 24.746 ms
7 * * *
8 4.14.98.38 (4.14.98.38) 50.381 ms 46.376 ms 43.220 ms
9 * * *
10 148.72.36.47 (148.72.36.47) 49.768 ms 42.868 ms 44.426 ms
11 148.72.36.59 (148.72.36.59) 42.996 ms 42.648 ms 41.981 ms
12 * * *
13 * * *
14 * * *
15 * * *
16 225.220.167.72.host.secureserver.net (72.167.220.225) 78.721 ms 43.946 ms 41.804 ms
