Join Log in
Score:1
ICT avatar Server

IPsec site-to-site VPN issues after recent Linux kernel update

ke flag
ICT

Last weekend we had an automatic security upgrade on one of our VPN gateways that connect sites to our cloud environment. After performing troubleshooting (via basic network troubleshooting e.g. via Wireshark) we identified one of the most recent security updates to be the cause of this. We have restored the system back to a known good state and have set (we believe to be) affected packages on hold.

It is a Ubuntu 20.04 LTS instance on AWS with linux-image-aws installed. We are using IPsec to connect several EdgeRouters to a private cloud environment.

After the upgrade all sites connect and communicate as usual, e.g. ICMP is working but we are unable to access certain services (such as RDP or SMB) in the private cloud environment.

The change logs for the related packages don't show any obvious linked change, so I am wondering if I am missing something fundamental. This configuration/setup has worked well for over a year now with no issues.

Known good version: linux-image-aws 5.8.0.1041.43~20.04.13

Problematic version: linux-image-aws 5.8.0.1042.44~20.04.14 and onwards (we have also tested latest 5.11 which seems to be affected)

IPsec configuration extract

# MAIN IPSEC VPN CONFIG
config setup

conn %default
        keyexchange=ikev1

# <REMOVED>
conn peer-rt1.<REMOVED>.net.au-tunnel-1
        left=%any
        right=rt1.<REMOVED>.net.au
        rightid="%any"
        leftsubnet=172.31.0.0/16
        rightsubnet=10.35.0.0/16
        ike=aes128-sha1-modp2048!
        keyexchange=ikev1
        ikelifetime=28800s
        esp=aes128-sha1-modp2048!
        keylife=3600s
        rekeymargin=540s
        type=tunnel
        compress=no
        authby=secret
        auto=route
        keyingtries=%forever
        dpddelay=30s
        dpdtimeout=120s
        dpdaction=restart

Thank you in advance.

EDIT 1: I have been able to capture another tcpdump from an unsuccessful RDP connection after another test upgrade, which looks like the following:

21:43:01.813502 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [S], seq 2706968963, win 64954, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:43:01.813596 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [S], seq 2706968963, win 64954, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:43:01.814238 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [S.], seq 152885333, ack 2706968964, win 64000, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0
21:43:01.839105 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1, win 1025, length 0
21:43:01.839168 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1, win 1025, length 0
21:43:01.840486 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 1:48, ack 1, win 1025, length 47
21:43:01.840541 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 1:48, ack 1, win 1025, length 47
21:43:01.843746 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 1:20, ack 48, win 63953, length 19
21:43:01.922120 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 20, win 1025, length 0
21:43:01.922212 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 20, win 1025, length 0
21:43:01.932646 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 48:226, ack 20, win 1025, length 178
21:43:01.932729 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 48:226, ack 20, win 1025, length 178
21:43:01.940677 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 20:1217, ack 226, win 63775, length 1197
21:43:01.967343 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 226:408, ack 1217, win 1020, length 182
21:43:01.967417 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 226:408, ack 1217, win 1020, length 182
21:43:01.969452 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 1217:1324, ack 408, win 63593, length 107
21:43:02.044376 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1324, win 1020, length 0
21:43:02.044471 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 1324, win 1020, length 0
21:43:02.135594 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 408:637, ack 1324, win 1020, length 229
21:43:02.135653 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [P.], seq 408:637, ack 1324, win 1020, length 229
21:43:02.136796 IP <REMOTE>.3389 > <LOCAL>.51099: Flags [P.], seq 1324:2609, ack 637, win 63364, length 1285
21:43:02.212871 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 2609, win 1025, length 0
21:43:02.212940 IP <LOCAL>.51099 > <REMOTE>.3389: Flags [.], ack 2609, win 1025, length 0
95
1 + 4
Tim avatar
gp flag
Tim
If you can't solve it, perhaps consider using an AWS VPN service rather than a server?
0
Reply
ICT avatar
ke flag
ICT
Hi Tim. That is a fair point, we have considered using VPC but wanted to retain some control over certain aspects that we only have via virtual machine based setup. But we will certainly consider it further if the issues persist.
0
Reply
cn flag
grasbueschel
Not sure how it relates to the package version, but MTU issues show similar patterns. Since ICMP is working, I'd try to increase ICMP packet size to see if there's a certain maximum size for packets reliably working and if so, would set the tunnel MTU manually (on both ends).
1
Reply
ICT avatar
ke flag
ICT
Hi grasbueschel, thanks for pointing this out - that sounds like it's worth trying. I will schedule this in as well!
0
Reply
Score:1
MLu avatar Server
id flag
MLu

Without digging into it - is it perhaps removal of the old, weak ciphers like sha1 and aes128? Try to change it to aes256-sha256-modp2048 in the working kernel version and then upgrade to see if it still breaks. Also maybe ikev2 instead of ikev1? But that’s a userspace concern, not a kernel setting.

Give that a try…

0 + 2
ICT avatar
ke flag
ICT
Thank you very much for the recommendation, I will give this a try after-hours!
0
Reply
ICT avatar
ke flag
ICT
Unfortunately this did not resolve the related issue. It looks like there is an issue with TCP traffic. I have updated my original post with a tcpdump of an unsuccessful RDP connection.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.