Score:0

Centos 7 Server not accepting traffic from the WAN

nl flag

A power outage required the shutdown of my CentOS 7 Mail & Web server. When the server came back up, a quick check showed no issues. The next morning, we noticed that we were not getting external email, but were getting internal email. The mail and web worked on the LAN but nothing from the Internet. Checked the logs, which had no external entries.

I rebooted my server in the event it did not come up correctly. Before the shutdown, my server was rebooted after an update and was working.

I tried an email test from a site that said it could not connect. Since my entire site was shutdown and brought back up, I looked at my network. I performed an nmap from my firewall and everything looked good.

I ran tcpdump on my firewall and observed traffic passing through my firewall to the server.

I ran tcpdump on my server and observed packets from the Internet.

08:46:23.975439 IP mta10.em.biglots.com.41216 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 978086259, win 29200, options [mss 1452,sackOK,TS val 660760064 ecr 0,nop,wscale 7], length 0
08:46:25.397998 IP mail9085.em1.tractorsupply.com.59682 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 3271108770, win 29200, options [mss 1380,sackOK,TS val 1306311584 ecr 0,nop,wscale 9], length 0
08:46:25.398141 IP mail9148.em9.tractorsupply.com.44498 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1954740324, win 29200, options [mss 1380,sackOK,TS val 1306311584 ecr 0,nop,wscale 9], length 0
08:46:28.860760 IP mta21.homedepotemail.com.49840 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1593758750, win 29200, options [mss 1452,sackOK,TS val 660843968 ecr 0,nop,wscale 7], length 0
08:46:31.987187 IP mta10.em.biglots.com.41216 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 978086259, win 29200, options [mss 1452,sackOK,TS val 660768080 ecr 0,nop,wscale 7], length 0
08:46:48.035676 IP mta10.em.biglots.com.41216 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 978086259, win 29200, options [mss 1452,sackOK,TS val 660784128 ecr 0,nop,wscale 7], length 0
08:46:55.147410 IP mta4.email.cbssports.com.34941 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 2074767968, win 29200, options [mss 1452,sackOK,TS val 671324357 ecr 0,nop,wscale 7], length 0
08:46:56.120471 IP mta4.email.cbssports.com.34941 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 2074767968, win 29200, options [mss 1452,sackOK,TS val 671325360 ecr 0,nop,wscale 7], length 0
08:46:58.124061 IP mta4.email.cbssports.com.34941 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 2074767968, win 29200, options [mss 1452,sackOK,TS val 671327364 ecr 0,nop,wscale 7], length 0
08:47:02.135673 IP mta4.email.cbssports.com.34941 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 2074767968, win 29200, options [mss 1452,sackOK,TS val 671331376 ecr 0,nop,wscale 7], length 0
08:47:10.187796 IP mta4.email.cbssports.com.34941 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 2074767968, win 29200, options [mss 1452,sackOK,TS val 671339392 ecr 0,nop,wscale 7], length 0
08:47:11.522665 IP p2-100094.mail.shape.com.33798 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 3054828880, win 14600, options [mss 1452], length 0
08:47:15.525548 IP p2-100094.mail.shape.com.33798 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 3054828880, win 14600, options [mss 1452], length 0
08:47:20.099434 IP mta10.em.biglots.com.41216 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 978086259, win 29200, options [mss 1452,sackOK,TS val 660816192 ecr 0,nop,wscale 7], length 0
08:47:23.523390 IP p2-100094.mail.shape.com.33798 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 3054828880, win 14600, options [mss 1452], length 0
08:47:26.184432 IP mta4.email.cbssports.com.34941 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 2074767968, win 29200, options [mss 1452,sackOK,TS val 671355424 ecr 0,nop,wscale 7], length 0
08:47:30.068282 IP mta.email-aaa.com.50888 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1962085054, win 29200, options [mss 1452,sackOK,TS val 671738399 ecr 0,nop,wscale 7], length 0
08:47:31.071586 IP mta.email-aaa.com.50888 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1962085054, win 29200, options [mss 1452,sackOK,TS val 671739402 ecr 0,nop,wscale 7], length 0
08:47:33.076189 IP mta.email-aaa.com.50888 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1962085054, win 29200, options [mss 1452,sackOK,TS val 671741408 ecr 0,nop,wscale 7], length 0
08:47:37.084162 IP mta.email-aaa.com.50888 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1962085054, win 29200, options [mss 1452,sackOK,TS val 671745416 ecr 0,nop,wscale 7], length 0
08:47:39.523364 IP p2-100094.mail.shape.com.33798 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 3054828880, win 14600, options [mss 1452], length 0

I stopped the firewall on my server.

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set blacklist src

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I checked /etc/hosts.allow and /etc/hosts.deny. Both were empty.

Does anyone have recommendations? Thank you!


What was the tcpdump command that you used to get this output?

tcpdump -i enp2s0 host 192.168.37.241 and dst port 25

Try it again without dst because that filtered out any reply traffic from your server.

tcpdump -i enp2s0 host 192.168.37.241 and port 25
11:12:45.602586 IP ms2.local.myexdomqqq.com.smtp > p1-101108.mail.westelm.com.52081: Flags [S.], seq 3798437961, ack 441006800, win 29200, options [mss 1460], length 0
11:12:46.618285 IP ms2.local.myexdomqqq.com.smtp > p1-101108.mail.westelm.com.52081: Flags [S.], seq 3798437961, ack 441006800, win 29200, options [mss 1460], length 0
11:12:47.602777 IP p1-101108.mail.westelm.com.52081 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 441006799, win 14600, options [mss 1452], length 0
11:12:47.602903 IP ms2.local.myexdomqqq.com.smtp > p1-101108.mail.westelm.com.52081: Flags [S.], seq 3798437961, ack 441006800, win 29200, options [mss 1460], length 0
11:12:48.272528 IP mail01.mscomm.morningstar.com.20305 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1779106704, win 29200, options [mss 1452,sackOK,TS val 3395772800 ecr 0,nop,wscale 7], length 0
11:12:48.272677 IP ms2.local.myexdomqqq.com.smtp > mail01.mscomm.morningstar.com.20305: Flags [S.], seq 1106233079, ack 1779106705, win 28960, options [mss 1460,sackOK,TS val 1783914 ecr 3395772800,nop,wscale 7], length 0
11:12:49.418204 IP ms2.local.myexdomqqq.com.smtp > mail01.mscomm.morningstar.com.20305: Flags [S.], seq 1106233079, ack 1779106705, win 28960, options [mss 1460,sackOK,TS val 1785060 ecr 3395772800,nop,wscale 7], length 0
11:12:49.618235 IP ms2.local.myexdomqqq.com.smtp > p1-101108.mail.westelm.com.52081: Flags [S.], seq 3798437961, ack 441006800, win 29200, opt^C165 packets captured

I ran traceroute and it made it to google.com

traceroute to google.com (142.250.72.46), 30 hops max, 60 byte packets
 1  _gateway (LLL.LLL.LLL.1)  0.340 ms  0.313 ms  0.339 ms
 2  192.168.0.1 (192.168.0.1)  1.014 ms  1.155 ms  1.529 ms
 3  albq-dsl-gw49.albq.qwest.net (67.42.200.49)  56.976 ms  58.320 ms  60.658 ms
 4  albq-agw1.inet.qwest.net (67.42.136.81)  60.770 ms  61.431 ms  61.419 ms
 5  205.171.210.9 (205.171.210.9)  71.650 ms  71.652 ms  71.462 ms
 6  72.14.219.162 (72.14.219.162)  72.970 ms 72.14.213.218 (72.14.213.218)  72.981 ms 72.14.219.162 (72.14.219.162)  72.660 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Did you check that blacklist you still have in your firewall?

I cleared the iptables

[root@mail ~]# systemctl stop firewalld
[root@mail ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set blacklist src

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

Still no mail. Outgoing mail works.

Michael Hampton avatar
cz flag
What was the tcpdump command that you used to get this output?
MadTechie avatar
nl flag
tcpdump -i enp2s0 host 192.168.37.241 and dst port 25
Michael Hampton avatar
cz flag
Try it again without `dst` because that filtered out any reply traffic from your server.
MadTechie avatar
nl flag
08:46:25.397998 IP mail9085.em1.tractorsupply.com.59682 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 3271108770, win 29200, options [mss 1380,sackOK,TS val 1306311584 ecr 0,nop,wscale 9], length 0 08:46:25.398141 IP mail9148.em9.tractorsupply.com.44498 > ms2.local.myexdomqqq.com.smtp: Flags [S], seq 1954740324, win 29200, options [mss 1380,sackOK,TS val 1306311584 ecr 0,nop,wscale 9], length 0
Michael Hampton avatar
cz flag
Did you check that blacklist you still have in your firewall?
MadTechie avatar
nl flag
traceroute -p 25 google.com 2 192.168.0.1 (192.168.0.1) 1.014 ms 1.155 ms 1.529 ms 3 albq-dsl-gw49.albq.qwest.net (67.42.200.49) 56.976 ms 58.320 ms 60.658 ms 4 albq-agw1.inet.qwest.net (67.42.136.81) 60.770 ms 61.431 ms 61.419 ms 5 205.171.210.9 (205.171.210.9) 71.650 ms 71.652 ms 71.462 ms 6 72.14.219.162 (72.14.219.162) 72.970 ms 72.14.213.218 (72.14.213.218) 72.981 ms 72.14.219.162 (72.14.219.162) 72.660 ms
Michael Hampton avatar
cz flag
You still have a blacklist ipset in your firewall that you did not actually remove. Again, you should check it.
MadTechie avatar
nl flag
I had rebooted the server with the firewall turned off. I clear them again.
Score:0
nl flag

Thanks to @MichaelHampton I found the issue. A few months ago, I added a backup WAN to my firewall and used it to move around my network connection equipment and when I finished, I did not remove the backup cable to my modem. When the firewall came up and got IP addresses, the backup had the primary IP and the primary got the secondary. I removed the cable and restarted the firewall and it works now. I knew it would be something simple and dumb.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.