Strange ovpn issue. No packets going through

Sorry for the long post but I put all information in.

First, my setup:

local network, protected by a pfSense box which hosts a Open-VPN server for my remote servers. Two server with public IPs are connected to my local network through the Ovpn connection. So two external servers connected each through Ovpn having access to my local network. Worked fine and smooth so far.

Second, IP setup:

local network 192.168.9.0/24, pfSense is 192.168.9.254 second local network 192.168.1.0/24 opvn network 10.150.2.0/24, pfsense has 10.150.2.1 server1 (with the issue) gets IP 10.150.2.3 (through ovpn dhcp) server2 (no issue) gets IP 10.150.2.2

Ususally I can get connection in both ways, no firewall involved within the ovpn network.

Third, issue description:

Now I realized one of the server is having issues with the ovpn connection. I can not ping the ovpn IP from my local network not get any other type of connection (as before was working fine).

Fourth, Troubleshooting steps:

The ovpn connection is established. Even though on server it shows state "UNKNOWN" (but I guess we can ignore this as it is the same state on the second server which runs fine).

root@netcup:~# ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   [...]
1: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
 [...]
   inet STRIPED/22 brd PUBLICIP scope global ens3
       valid_lft forever preferred_lft forever
    inet6 STRIPED
[...]
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none 
    inet 10.150.2.3/24 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::caf:4f24:6589:6c01/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

The log on the server tells me the connection is established fine. (sorry for long logfile, loglevel 6):

Jan 29 13:46:38 nc ovpn-router-UDP4-1194-nc[12375]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
Jan 29 13:46:38 nc ovpn-router-UDP4-1194-nc[12375]: [router.knebb.de] Peer Connection Initiated with [AF_INET]80.139.244.247:1194
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: SENT CONTROL [router.knebb.de]: 'PUSH_REQUEST' (status=1)
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: UDP WRITE [89] to [AF_INET]80.139.244.247:1194: P_CONTROL_V1 kid=0 pid=[ #15 ] [ ] pid=7 DATA len=35
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: UDP READ [62] from [AF_INET]80.139.244.247:1194: P_ACK_V1 kid=0 pid=[ #13 ] [ 7 ]
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: UDP READ [427] from [AF_INET]80.139.244.247:1194: P_CONTROL_V1 kid=0 pid=[ #14 ] [ ] pid=8 DATA len=373
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: PUSH: Received control message: 'PUSH_REPLY,route 192.168.9.0 255.255.255.0,route 192.168.1.0 255.255.255.0,dhcp-option DOMAIN knebb.de,dhcp-option DNS 192.168.9.254,dhcp-option DNS 192.168.1.254,dhcp-option NTP 192.168.9.254,dhcp-option NTP 192.168.1.254,route-gateway 10.150.2.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.150.2.3 255.255.255.0,peer-id 1,cipher AES-128-CBC'
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: timers and/or timeouts modified
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: --ifconfig/up options modified
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: route options modified
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: route-related options modified
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: peer-id set
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: adjusting link_mtu to 1624
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: OPTIONS IMPORT: data channel crypto options modified
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: Outgoing Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: Outgoing Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: Incoming Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: Incoming Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_route_v4_best_gw query: dst 0.0.0.0
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: checking for received messages
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: rtnl: received 532 bytes
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_route_v4_best_gw result: via PUBLICIP dev ens3
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: ROUTE_GATEWAY PUBLICIP/255.255.252.0 IFACE=ens3 HWADDR=1a:bd:a8:dc:00:5b
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: TUN/TAP device tun0 opened
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: do_ifconfig, ipv4=1, ipv6=0
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_iface_mtu_set: mtu 1500 for tun0
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: checking for received messages
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: rtnl: received 36 bytes
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_iface_up: set tun0 up
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: checking for received messages
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: rtnl: received 36 bytes
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_addr_v4_add: 10.150.2.3/24 dev tun0
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: checking for received messages
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: rtnl: received 36 bytes
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_route_v4_add: 192.168.9.0/24 via 10.150.2.1 dev [NULL] table 0 metric -1
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: checking for received messages
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: rtnl: received 36 bytes
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: net_route_v4_add: 192.168.1.0/24 via 10.150.2.1 dev [NULL] table 0 metric -1
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: checking for received messages
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: sitnl_send: rtnl: received 36 bytes
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Jan 29 13:46:40 nc ovpn-router-UDP4-1194-nc[12375]: Initialization Sequence Completed

So the connection appears to be valid. But I still have the issue. Routing is fine:

root@nc:~# ip r sh
default via PUBLICGW dev ens3 onlink 
10.150.2.0/24 dev tun0 proto kernel scope link src 10.150.2.3 
PUBLICIP/22 dev ens3 proto kernel scope link src PUBLICIP 
192.168.1.0/24 via 10.150.2.1 dev tun0 
192.168.9.0/24 via 10.150.2.1 dev tun0 

There are no firewall rules involved:

root@nc:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Now I have the issue my server1 is not responding to any network packets. Trying to ping does not succeed:

user@mac ~ % ping nc
PING nc.knebb.de (10.150.2.3): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

A tcpdump at the pfsense router shows it sends the packets out to the server but there are not packets coming back from the server:

14:06:19.472017 IP 192.168.9.50 > 10.150.2.3: ICMP echo request, id 20042, seq 0, length 64
14:06:20.478678 IP 192.168.9.50 > 10.150.2.3: ICMP echo request, id 20042, seq 1, length 64
14:06:21.482551 IP 192.168.9.50 > 10.150.2.3: ICMP echo request, id 20042, seq 2, length 64

Checking on the server1 I only see the "alive-packets" from ovpn:

root@netcup:~# tcpdump -i ens3 port 1194
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:10:16.596788 IP p508.dip0.t-ipconnect.de.openvpn > nc.knebb.de.openvpn: UDP, length 84
14:10:16.597181 IP nc.knebb.de.openvpn > p508.dip0.t-ipconnect.de.openvpn: UDP, length 84

Doing tcpdump on tun0 does not show any packets!

Last resort: reboot:

Now whatever I do, the issue persists. systemctl restart ovpn...does not help. ovpn end existing connection and reestablishes the new one with the same issue.

Once rebooted, the vpn connection runs initially fine. I can ping. But shortly after it gets into the same state as before where the connection only runs from time to time::

user@mac ~ % ping nc
PING nc.knebb.de (10.150.2.3): 56 data bytes
64 bytes from 10.150.2.3: icmp_seq=0 ttl=63 time=282.514 ms
Request timeout for icmp_seq 1
64 bytes from 10.150.2.3: icmp_seq=2 ttl=63 time=545.166 ms
64 bytes from 10.150.2.3: icmp_seq=3 ttl=63 time=39.950 ms
Request timeout for icmp_seq 4
[...]
Request timeout for icmp_seq 83
Request timeout for icmp_seq 84
64 bytes from 10.150.2.3: icmp_seq=85 ttl=63 time=54.630 ms
64 bytes from 10.150.2.3: icmp_seq=86 ttl=63 time=78.752 ms
[...]
64 bytes from 10.150.2.3: icmp_seq=103 ttl=63 time=356.079 ms
Request timeout for icmp_seq 104
Request timeout for icmp_seq 105

Anyone of you guys having an idea what is wrong here?

THANKS!

/KNEBB