Join Log in
Score:0
avatar Server

NAT pre/output/post routing/forwarding fails a few minutes after a reboot. What would ask the firewall to stop forwarding?

cn flag
Alexis Wilke

This morning (May 5, 2023), I found that my LAN stopped working on my Ubuntu 18.04 installation. All computers are locked out because the main server FORWARD rules are being ignored.

When I first start the computer, it works as expected. Nothing was installed (one of the first things I checked) and the forwarding has been working for like 10 years. I did not change my firewall. No installation... Just after a reboot it works for a minute or two, I think that is the time it takes for something to start and mess that up. But I would need to know where to look to determine what is causing the issue (hopefully not code from a hacker).

Below, we can see that the PREROUTING, INPUT, and POSTROUTING worked for a little bit. Now the counters are stuck at those numbers. No more packets are going through the NAT table, except the INPUT chain. What I really need, though, is the POSTROUTING chain which has a SNAT and thus generates a forward. (I also have a MASQUERADE, but that is not required, it is ignored because of the SNAT rule. I just added the MASQUERADE just in case it would help.)

Chain PREROUTING (policy ACCEPT 642 packets, 43832 bytes)
pkts    bytes target     prot opt in     out     source               destination         
 208    13747 LOG        all  --  *      *       192.168.3.183        0.0.0.0/0            LOG flags 0 level 4

Chain INPUT (policy ACCEPT 825 packets, 51359 bytes)
pkts    bytes target     prot opt in     out     source               destination         
 189     9873 LOG        all  --  *      *       192.168.0.18        0.0.0.0/0            LOG flags 0 level 4

Chain OUTPUT (policy ACCEPT 120 packets, 13320 bytes)
pkts    bytes target     prot opt in     out     source               destination         
   0        0 LOG        all  --  *      *       192.168.0.18        0.0.0.0/0            LOG flags 0 level 4

Chain POSTROUTING (policy ACCEPT 89 packets, 8970 bytes)
pkts    bytes target     prot opt in     out     source               destination         
 153    10268 LOG        all  --  *      *       192.168.0.18        0.0.0.0/0            LOG flags 0 level 4
 153    10268 SNAT       all  --  *      eno1    192.168.0.18        0.0.0.0/0            to:10.0.0.2

I show the SNAT rule. As we can see, it received the same number of packets and my MS-Windows box said that it could see the Internet for a little while. So I'm pretty sure that the network is properly setup on a reboot.

Things I've looked into already (outside of the iptables above):

$ sudo sysctl  net.ipv4.ip_forward
net.ipv4.ip_forward = 1

$ sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1

I also checked conntrack, but there is nothing in particular I can see there, except the "UNREPLIED" which I would imagine is a different issue?

udp      17 16 src=192.168.0.18 dst=75.75.75.75 sport=56804 dport=53 ...
            ... [UNREPLIED] src=75.75.75.75 dst=192.168.0.18 sport=53 dport=56804 mark=0 use=1

Note: it is one long line, cut in half here to make it easier to read.

Also I have a docker from snap, although the daemon is running, the DOCKER rules that often appear in the firewall are not there at the moment.

Btw, the LAN itself works. I have various things that communicate with the main server or between computers and it works just fine.

The bridge addresses:

$ ip -4 -br address
lo               UNKNOWN        127.0.0.1/8 
eno1             UP             x.x.x.x/30 192.168.1.1/24 10.1.10.2/24 
eno2             UP             192.168.0.45/24 192.168.2.1/24 192.168.3.1/24 172.16.1.1/24 10.0.2.10/24 10.0.3.10/24 
docker0          DOWN           172.17.0.1/16

And the routes:

$ ip route
default via 96.67.192.226 dev eno1 proto static 
10.0.2.0/24 dev eno2 proto kernel scope link src 10.0.2.10 
10.0.3.0/24 dev eno2 proto kernel scope link src 10.0.3.10 
10.1.10.0/24 dev eno1 proto kernel scope link src 10.1.10.2 
x.x.x.x/30 dev eno1 proto kernel scope link src x.x.x.x 
172.16.1.0/24 dev eno2 proto kernel scope link src 172.16.1.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.0.0/24 dev eno2 proto kernel scope link src 192.168.0.45 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev eno2 proto kernel scope link src 192.168.2.1 
192.168.3.0/24 dev eno2 proto kernel scope link src 192.168.3.1

The x.x.x.x is a static IP address.

Docker counter measure for forwarding packets:

Chain DOCKER-USER (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 early_forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Resolution: I upgraded to 20.04 (and will move on to 22.04 very soon). I just cannot explain why/how the 18.04 iptables would break in this way, but it really did for me. 20.04 is holding so far.

49
0 + 6
A.B avatar
cl flag
A.B
Are you sure Docker doesn't create conflicting LANs? What's the output of `ip -4 -br address; ip route`?
0
Reply
cn flag
Alexis Wilke
@A.B Well... when I do a `ps -ef | grep docker`, it shows up only if it is running and at that point I see rules in my firewall. However, after a reboot and no use of docker, the daemon is not running. Also I noticed what looks like a crashed process. The name is [`SEPDRV_ABNORMAL` and it is a kernel thread](https://unix.stackexchange.com/questions/746200/what-is-the-process-named-sepdrv-abnormal-under-linux). Could that be related?
0
Reply
A.B avatar
cl flag
A.B
(you should still obfuscate public addresses). I don't see anything obvious. You could keep a few logs running right at boot. For example: `ip -ts monitor > /tmp/iproute2.log` + `nft monitor 2>&1 | ts > /tmp/nft.log` + (for the iptables part only, even if the previous command would be enough) `stdbuf -oL xtables-monitor -e | ts > /tmp/iptables.log`
0
Reply
A.B avatar
cl flag
A.B
Ah again about Docker. Docker does change the FORWARD policy to DROP, but you're aware of this right, since you're using Docker, you must have adapted your rules adequately?
0
Reply
cn flag
Alexis Wilke
I added a quick update with a rule that Docker understands and runs before its DROP rules. This allows my FORWARD rules to happen. As we can see, the counters are 0 meaning that it never reaches that rule (because Docker has not be started).
0
Reply
cn flag
Alexis Wilke
@A.B Well... since I really couldn't find anything close to a working solution, I decided to upgrade, since I was still on 18.04... and now that I'm on 20.04, it works. So something in 18.04 broke... The new version is holding, it passed the night and my LAN functions as before (so far).
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.