Join Log in
Score:2
oliverplaner avatar Server

Linux host routing VLAN traffic cannot receive or make network connections

bs flag
oliverplaner

I have a CentOS 7 system that has three interfaces. I recently converted one on those interfaces to using VLANs. This system servers as a firewall and router between the various networks. Everything seemed to be working as expected, except one item. I cannot connect into or out of the CentOS system from/to any hosts on the VLANs.

For example, if I ssh from 192.168.32.95 (a host on VLAN 32) to 192.168.32.1 (VLAN 32 interface on the CentOS host), I can see the packets with tcpdump arrive on the enp3s0 interface, tagged with vlan 32. The iptables INPUT chain logs and ACCEPTS the packet. After that, the packet is gone. No SYN-ACK sent, no RST sent.

Similarly, for something like NTP from 192.168.32.1 to 192.168.32.3, the firewall logs and ACCEPTs the outgoing NTP packet, but then no packet is seen by tcpdump leaving on enp3s0 (i.e. enp3s0.32)

Pings from VLAN 32 work...I can ping the CentOS host from VLAN 32 and get a reply.

The CentOS system is forwarding packets between all combinations of the VLAN and non-VLAN interfaces.

Any thoughts on what could account for these connection issue, or even a way to further track down what is happening?

# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether c0:25:e9:0e:cb:a1 brd ff:ff:ff:ff:ff:ff
    inet 10.20.30.177/28 brd 10.20.30.191 scope global enp1s0
       valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 84:16:f9:05:3a:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.16.1/24 brd 192.168.16.255 scope global enp2s0
       valid_lft forever preferred_lft forever
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 6c:f0:49:42:5b:fa brd ff:ff:ff:ff:ff:ff
5: enp3s0.32@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6c:f0:49:42:5b:fa brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.1/24 brd 192.168.32.255 scope global enp3s0.32
       valid_lft forever preferred_lft forever
6: enp3s0.50@enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6c:f0:49:42:5b:fa brd ff:ff:ff:ff:ff:ff
    inet 192.168.50.1/24 brd 192.168.50.255 scope global enp3s0.50
       valid_lft forever preferred_lft forever

Here is the INPUT chain where I put a logging and ACCEPT rule right at the beginning.

# iptables -t filter -S INPUT
-P INPUT DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 192.168.32.95/32 -d 192.168.32.1/32 -p tcp -m tcp --dport 22 -j LOG --log-prefix "TEST IN: "
-A INPUT -s 192.168.32.95/32 -d 192.168.32.1/32 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -m state --state INVALID -j INVALID
-A INPUT -m state --state UNTRACKED -j UNTRACKED
-A INPUT -i enp1s0 -j INPUT-EXT
-A INPUT -i enp2s0 -j INPUT-MID
-A INPUT -i enp3s0.32 -j INPUT-V32
-A INPUT -i enp3s0.50 -j INPUT-V50
-A INPUT -j LOG --log-prefix "DROP EOC INPUT: " --log-tcp-options --log-ip-options
-A INPUT -j DROP

Firewall logs the SYN packet.

Jul  4 14:21:49 outside kernel: [112380.489332] TEST IN: IN=enp3s0.32 OUT= MAC=6c:f0:49:42:5b:fa:52:54:00:35:f4:e4:08:00:45:00:00:3c SRC=192.168.32.95 DST=192.168.32.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=17731 DF PROTO=TCP SPT=51926 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

Output of tcpdump shows multiple copies of the SYN packet arriving.

# tcpdump -e -nn -i enp3s0 host 192.168.32.95 and host 192.168.32.1
14:21:49.516122 52:54:00:35:f4:e4 > 6c:f0:49:42:5b:fa, ethertype 802.1Q (0x8100), length 78: vlan 32, p 0, ethertype IPv4, 192.168.32.95.51926 > 192.168.32.1.22: Flags [S], seq 3631520692, win 29200, options [mss 1460,sackOK,TS val 69303507 ecr 0,nop,wscale 7], length 0
14:21:50.529043 52:54:00:35:f4:e4 > 6c:f0:49:42:5b:fa, ethertype 802.1Q (0x8100), length 78: vlan 32, p 0, ethertype IPv4, 192.168.32.95.51926 > 192.168.32.1.22: Flags [S], seq 3631520692, win 29200, options [mss 1460,sackOK,TS val 69303760 ecr 0,nop,wscale 7], length 0
14:21:52.543926 52:54:00:35:f4:e4 > 6c:f0:49:42:5b:fa, ethertype 802.1Q (0x8100), length 78: vlan 32, p 0, ethertype IPv4, 192.168.32.95.51926 > 192.168.32.1.22: Flags [S], seq 3631520692, win 29200, options [mss 1460,sackOK,TS val 69304264 ecr 0,nop,wscale 7], length 0
14:21:56.607938 52:54:00:35:f4:e4 > 6c:f0:49:42:5b:fa, ethertype 802.1Q (0x8100), length 78: vlan 32, p 0, ethertype IPv4, 192.168.32.95.51926 > 192.168.32.1.22: Flags [S], seq 3631520692, win 29200, options [mss 1460,sackOK,TS val 69305280 ecr 0,nop,wscale 7], length 0
14:22:04.799936 52:54:00:35:f4:e4 > 6c:f0:49:42:5b:fa, ethertype 802.1Q (0x8100), length 78: vlan 32, p 0, ethertype IPv4, 192.168.32.95.51926 > 192.168.32.1.22: Flags [S], seq 3631520692, win 29200, options [mss 1460,sackOK,TS val 69307328 ecr 0,nop,wscale 7], length 0

Additional requested information:

# ip route
default via 50.196.138.190 dev enp1s0
10.20.30.176/28 dev enp1s0 proto kernel scope link src 10.20.30.177
192.168.16.0/24 dev enp2s0 proto kernel scope link src 192.168.16.1
192.168.32.0/24 dev enp3s0.32 proto kernel scope link src 192.168.32.1
192.168.50.0/24 dev enp3s0.50 proto kernel scope link src 192.168.50.1

# ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# ip route show table local
broadcast 10.20.30.176 dev enp1s0 proto kernel scope link src 10.20.30.177
local 10.20.30.177 dev enp1s0 proto kernel scope host src 10.20.30.177
broadcast 10.20.30.191 dev enp1s0 proto kernel scope link src 10.20.30.177
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 192.168.16.0 dev enp2s0 proto kernel scope link src 192.168.16.1
local 192.168.16.1 dev enp2s0 proto kernel scope host src 192.168.16.1
broadcast 192.168.16.255 dev enp2s0 proto kernel scope link src 192.168.16.1
broadcast 192.168.32.0 dev enp3s0.32 proto kernel scope link src 192.168.32.1
local 192.168.32.1 dev enp3s0.32 proto kernel scope host src 192.168.32.1
broadcast 192.168.32.255 dev enp3s0.32 proto kernel scope link src 192.168.32.1
broadcast 192.168.50.0 dev enp3s0.50 proto kernel scope link src 192.168.50.1
local 192.168.50.1 dev enp3s0.50 proto kernel scope host src 192.168.50.1
broadcast 192.168.50.255 dev enp3s0.50 proto kernel scope link src 192.168.50.1

I had added a specific logging and ACCEPT rule in the OUTPUT chain, but nothing ever got logged from it.

# iptables -S OUTPUT
-P OUTPUT DROP
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -s 192.168.32.1/32 -d 192.168.32.95/32 -p tcp -m tcp --sport 22 -j LOG --log-prefix "TEST OUT: "
-A OUTPUT -s 192.168.32.1/32 -d 192.168.32.95/32 -p tcp -m tcp --sport 22 -j ACCEPT
-A OUTPUT -m state --state INVALID -j INVALID
-A OUTPUT -m state --state UNTRACKED -j UNTRACKED
-A OUTPUT -o enp1s0 -j OUTPUT-EXT
-A OUTPUT -o enp2s0 -j OUTPUT-MID
-A OUTPUT -o enp3s0.32 -j OUTPUT-V32
-A OUTPUT -o enp3s0.50 -j OUTPUT-V50
-A OUTPUT -j LOG --log-prefix "DROP EOC OUTPUT: " --log-tcp-options --log-ip-options
-A OUTPUT -j DROP

# ss -atn
State      Recv-Q Send-Q                Local Address:Port        Peer Address:Port
LISTEN     0      128                       127.0.0.1:199                    *:*
LISTEN     0      128                               *:22                     *:*
SYN-RECV   0      0          192.168.32.1%if378475780:22         192.168.32.95:36778
LISTEN     0      100                       127.0.0.1:25                     *:*
ESTAB      0      292                    192.168.16.1:22         192.168.16.61:49618


# cat /etc/sysconfig/network-scripts/ifcfg-enp3s0.32
DEVICE=enp3s0.32
TYPE=Ethernet
ONBOOT=yes
VLAN=yes
BOOTPROTO=static
IPADDR=192.168.32.1
NETMASK=255.255.255.0
IPV6INIT=no

# cat /etc/sysconfig/network-scripts/ifcfg-enp3s0
DEVICE=enp3s0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPV6INIT=no

Here are three attempts to ssh into this host. Each listing below is after the previous ssh has has exited with Connection timed out and after the previous SYN-RECV is gone. I removed the LISTEN and ESTAB lines from the out as they are essentially the same as the previous output. Also, the SYN-RECV lines were blank in the process column.

# ss -atnp
State      Recv-Q Send-Q              Local Address:Port   Peer Address:Port
SYN-RECV   0      0            192.168.32.1%if-8096:22    192.168.32.95:47290

# ss -atnp
State      Recv-Q Send-Q              Local Address:Port   Peer Address:Port
SYN-RECV   0      0        192.168.32.1%if378479604:22    192.168.32.95:47440

# ss -atnp
State      Recv-Q Send-Q              Local Address:Port   Peer Address:Port
SYN-RECV   0      0        192.168.32.1%if378479648:22    192.168.32.95:47580

Another experiment, I tried starting multple ssh connections, 1 seconds apart:

for x in 1 2 3 4 5 6 7 8 9 10 ; do ssh -n 192.168.32.1 date & sleep 1; done

and after all the ssh process are running, the ss output was as below. No SYN-ACK packets were reported by tcpdump. ss output stayed the same until connections timed out.

# ss -atn
State       Recv-Q Send-Q             Local Address:Port   Peer Address:Port
SYN-RECV    0      0       192.168.32.1%if378480635:22    192.168.32.95:50492
SYN-RECV    0      0       192.168.32.1%if378480636:22    192.168.32.95:50502
SYN-RECV    0      0       192.168.32.1%if378480636:22    192.168.32.95:50498
SYN-RECV    0      0       192.168.32.1%if378480635:22    192.168.32.95:50494
SYN-RECV    0      0                   192.168.32.1:22    192.168.32.95:50488
SYN-RECV    0      0       192.168.32.1%if378480634:22    192.168.32.95:50486
SYN-RECV    0      0       192.168.32.1%if378480634:22    192.168.32.95:50490
SYN-RECV    0      0                   192.168.32.1:22    192.168.32.95:50504
SYN-RECV    0      0       192.168.32.1%if378480636:22    192.168.32.95:50496
SYN-RECV    0      0       192.168.32.1%if378480636:22    192.168.32.95:50500

The system setup is pretty standard configuration. Prior to the recent change, the enp3s0 interface was the 192.168.32.0/24 network. The only real changes were the ifcfg-enp3s0* files under sysconfig, and updating the firewall rules. Most of the hosts moved unchanged from the old subnet to vlan 32.

114
0 + 4
A.B avatar
cl flag
A.B
Can you also provide routes? `ip route; ip rule`? And also outgoing iptables rules (filter/OUTPUT etc.). When running `ss -atn | grep ^SYN` multiple times, is there any SYN-RECV right at the time the log happened?
1
Reply
A.B avatar
cl flag
A.B
1/ Input is working, SYN-RECV means SYN was received, the problem is in output 2/ What is the interface `if378475780`, why is it not present in the output of `ip addr show`? And why would it be used at all? it's not described anywhere. Are you using LD_PRELOAD mechanisms on sshd? I don't think it has an option to bind to a device (SO_BINDTODEVICE). Is that even the same sshd process? try again `ss` but also add the `-p` option. Of course add all useful information you appear to have omitted.
0
Reply
A.B avatar
cl flag
A.B
I'm very surprised by the output of the `ss` command but as I can't explain it... sorry.
0
Reply
djdomi avatar
za flag
djdomi
i read this page daily and daily i learn a lot more commands for myself beeing useful +1 for ss suggestion even its a bad term :)
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.