Score:0

OpenVPN setup inexplicably stopped working

vu flag

After some struggle a couple months ago, I set up an openVPN server and client to remotely connect to my first PC build. Everything is running Ubuntu 20. It worked fine until now, allowing me to connect from every remote location I have tried. I used this tutorial from Digital Ocean to set things up initially. I now have TLS issues and have no idea how to proceed so I'm posting here.

I didn't explicitly touch anything related to openVPN, or do any other massive installs on my server machine, but I can no longer connect to it from my laptop client. I tried to set up a second laptop client to see if maybe it was a client side error, but the second laptop didn't work either. I then uninstalled and reinstalled openvpn on both ends and created new keys and everything from scratch. I am still getting the same TLS Handshake error shown below in this client output.

Clue 1: As Nikita pointed out, the server output doesn't show any client tried to connect. In the past, I have seen attempts in the server output. After issuing the openvpn *conf commands on server and client, I issued some tcpdump commands (though I know little about it) on the server

~$ sudo tcpdump -D
[sudo] password for adnan: 
1.enp5s0 [Up, Running]
2.tun0 [Up, Running]
3.lo [Up, Running, Loopback]
4.any (Pseudo-device that captures on all interfaces) [Up, Running]
5.wlo1 [Up]
6.docker0 [Up]
7.br-d2c78a773ae5 [Up]
8.br-4b07fa21428c [Up]
9.bluetooth-monitor (Bluetooth Linux Monitor) [none]
10.nflog (Linux netfilter log (NFLOG) interface) [none]
11.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
12.bluetooth0 (Bluetooth adapter number 0) [none]
~$ sudo tcpdump -i tun0 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes

and the client

~$ sudo tcpdump -D
1.wlo1 [Up, Running]
2.lo [Up, Running, Loopback]
3.any (Pseudo-device that captures on all interfaces) [Up, Running]
4.docker0 [Up]
5.br-4fe775d77579 [Up]
6.bluetooth-monitor (Bluetooth Linux Monitor) [none]
7.nflog (Linux netfilter log (NFLOG) interface) [none]
8.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
9.bluetooth0 (Bluetooth adapter number 0) [none]

Clue 2?: One thing that I did notice when going through the tutorial again was that the output of ip route list default seemed to have changed from enp4s0 to enp5s0, but I don't know if this is relevant.

Clue 3?: The above tutorial suggests running systemd-resolve --status tun0 on the client side but it returns Failed to resolve interface "tun0", ignoring: No such device. But I don't know how seriously to take this... On further reading, I guess this would only be relevant if I am trying to push all traffic through the VPN, which I am not. So maybe this is irrelevant.

The client output is

client$ openvpn laptop_client.conf
Sun Jan 29 08:12:29 2023 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Mar 22 2022
Sun Jan 29 08:12:29 2023 library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Sun Jan 29 08:12:29 2023 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sun Jan 29 08:12:29 2023 Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Sun Jan 29 08:12:29 2023 Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Sun Jan 29 08:12:29 2023 Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Sun Jan 29 08:12:29 2023 Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Sun Jan 29 08:12:29 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]140.141.196.45:11111
Sun Jan 29 08:12:29 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Sun Jan 29 08:12:29 2023 UDP link local: (not bound)
Sun Jan 29 08:12:29 2023 UDP link remote: [AF_INET]140.141.196.45:11111
Sun Jan 29 08:12:29 2023 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Sun Jan 29 08:13:29 2023 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Sun Jan 29 08:13:29 2023 TLS Error: TLS handshake failed
Sun Jan 29 08:13:29 2023 SIGUSR1[soft,tls-error] received, process restarting
Sun Jan 29 08:13:29 2023 Restart pause, 5 second(s)
Sun Jan 29 08:13:34 2023 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sun Jan 29 08:13:34 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]140.141.196.45:11111
Sun Jan 29 08:13:34 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Sun Jan 29 08:13:34 2023 UDP link local: (not bound)
Sun Jan 29 08:13:34 2023 UDP link remote: [AF_INET]140.141.196.45:11111
Sun Jan 29 08:14:34 2023 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Sun Jan 29 08:14:34 2023 TLS Error: TLS handshake failed
Sun Jan 29 08:14:34 2023 SIGUSR1[soft,tls-error] received, process restarting
Sun Jan 29 08:14:34 2023 Restart pause, 5 second(s)
Sun Jan 29 08:14:39 2023 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sun Jan 29 08:14:39 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]140.141.196.45:11111
Sun Jan 29 08:14:39 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Sun Jan 29 08:14:39 2023 UDP link local: (not bound)
Sun Jan 29 08:14:39 2023 UDP link remote: [AF_INET]140.141.196.45:11111

The server output is

root@build1:/etc/openvpn/server# openvpn server_build1.conf 
Sat Jan 28 21:48:58 2023 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Mar 22 2022
Sat Jan 28 21:48:58 2023 library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Sat Jan 28 21:48:58 2023 Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Sat Jan 28 21:48:58 2023 Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Sat Jan 28 21:48:58 2023 Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
Sat Jan 28 21:48:58 2023 Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authentication
Sat Jan 28 21:48:58 2023 ROUTE_GATEWAY 10.1.2.1/255.255.255.0 IFACE=enp5s0 HWADDR=d8:bb:c1:d9:d3:33
Sat Jan 28 21:48:58 2023 TUN/TAP device tun0 opened
Sat Jan 28 21:48:58 2023 TUN/TAP TX queue length set to 100
Sat Jan 28 21:48:58 2023 /sbin/ip link set dev tun0 up mtu 1500
Sat Jan 28 21:48:58 2023 /sbin/ip addr add dev tun0 local 10.8.1.1 peer 10.8.1.2
Sat Jan 28 21:48:58 2023 /sbin/ip route add 10.8.1.0/24 via 10.8.1.2
Sat Jan 28 21:48:58 2023 Could not determine IPv4/IPv6 protocol. Using AF_INET
Sat Jan 28 21:48:58 2023 Socket Buffers: R=[212992->212992] S=[212992->212992]
Sat Jan 28 21:48:58 2023 UDPv4 link local (bound): [AF_INET][undef]:11111
Sat Jan 28 21:48:58 2023 UDPv4 link remote: [AF_UNSPEC]
Sat Jan 28 21:48:58 2023 GID set to nogroup
Sat Jan 28 21:48:58 2023 UID set to nobody
Sat Jan 28 21:48:58 2023 MULTI: multi_init called, r=256 v=256
Sat Jan 28 21:48:58 2023 IFCONFIG POOL: base=10.8.1.4 size=62, ipv6=0
Sat Jan 28 21:48:58 2023 IFCONFIG POOL LIST
Sat Jan 28 21:48:58 2023 Initialization Sequence Completed

The laptop_client.conf file has in it (I've redacted things that I think I'm supposed to)

client
dev tun
proto udp
remote REDACTED 11111
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
remote-cert-tls server
cipher AES-256-GCM
auth SHA256
verb 3
key-direction 1
script-security 2
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre
dhcp-option DOMAIN-ROUTE .
<ca>
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----
</ca>
<cert>
Certificate:
    Data:
        REDACTED
...
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
REDACTED
-----END PRIVATE KEY-----
</key>
<tls-crypt>
-----BEGIN OpenVPN Static key V1-----
REDACTED
-----END OpenVPN Static key V1-----
</tls-crypt>

and the server_build1.conf file is

port 11111
proto udp
dev tun
ca ca.crt
cert server_build1.crt
key server_build1.key  # This file should be kept secret
dh none
server 10.8.1.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
push "route 10.1.2.0 255.255.255.0"
keepalive 10 120
tls-crypt ta.key
cipher AES-256-GCM
auth SHA256
user nobody
group nogroup
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
verb 3
explicit-exit-notify 1

In case it's relevant, the server firewall seems to be working

root@build1:/etc/openvpn/server# ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere                  
11111/udp                  ALLOW       Anywhere                  
5900/tcp                   ALLOW       Anywhere                  
OpenSSH (v6)               ALLOW       Anywhere (v6)             
11111/udp (v6)             ALLOW       Anywhere (v6)             
5900/tcp (v6)              ALLOW       Anywhere (v6) 
Nikita Kipriyanov avatar
za flag
This still looks like connection doesn't reach the server at all. While that's evident enough for me from the server log file (it doesn't log anything related to the client connection attempts), you can confirm this by capturing traffic simultaneously on the server and a client with tcpdump or wireshark or whatever and check that all packets sent by one side were received by the peer. Maybe not any of your firewalls, but some system in between is involved in this.
MaanDoabeDa avatar
vu flag
@Nikita That seems to make sense. In the past I have seen the server react in its output when the client tries to connect, and that is not happening anymore. I'll add some output from `tcpdump` though I've never really used it before. Any thoughts on what kind of system could be getting in between here?
Nikita Kipriyanov avatar
za flag
Aaaaaand... this is not something we can help or answer here. Sorry. I'd even say it is off topic. And it could be any system. Try another port. Try TCP (just for the case, to see if it works).
MaanDoabeDa avatar
vu flag
@NikitaKipriyanov Would you perhaps recommend setting up the openVPN server in docker, so as to prevent these sorts of random issues?
Nikita Kipriyanov avatar
za flag
If the cause is really external to your systems, the problem is completely independent of how you run the software. If will not avoid nor exaggerate it. I already gave my recommendations.
MaanDoabeDa avatar
vu flag
@NikitaKipriyanov Thanks for the tips! You have a strong intuition for this sort of thing, it seems. I thought that docker thoroughly isolated things from the rest of the computer, but on second thought, that seems sort of impossible. So I didn't waste much time trying out docker because of your recommendation. I instead tried opening up a port for `ssh` and that also hung. After that I finally realized the issue was with my router and `no-ip` communicating, as I mention in my answer.
Score:0
vu flag

This answer is unlikely to be helpful to others but the issue was related to the fact that I am using no-ip.com to give me a domain name that follows my dynamic IP. I changed my password on my account because I forgot it, but then forgot that my router needs the password to be able to communicate with no-ip. Even after changing the password back on no-ip, the IP wasn't updating on no-ip so I had to go click a few buttons on the website to get it to update. Then everything started working!

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.