Join Log in
Score:1
avatar Ubuntu

TLS Negotiation fails from Multipass VM but succeeds from host machine (or other machines on network)

ke flag
scubbo

Summary

Running curl -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt from a Multipass VM appears to result in a hung TLS negotiation, as there is no (IN), TLS handshake, Server hello (2) response for the 302-redirected URL.

Detail

I'm trying to install k3s on a Multipass VM on my Mac, as documented here and here. The k3s installation process involves downloading and running an installation script (curl -sfL https://get.k3s.io | sh -) - however, this script hangs partway through when run on the Multipass VM:

[INFO]  Finding release for channel stable
[INFO]  Using v1.25.4+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
[...hangs...]

From stepping through the script, I found that it's hanging on a call to curl -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt. I tried to replicate this call directly (with verbose flags set) - selected output below:

$ curl -V
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

$ curl -vvv -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
*   Trying 192.30.255.112:443...
* Connected to github.com (192.30.255.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
[...]
< HTTP/2 302
[...]
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T222055Z&X-Amz-Expires=300&X-Amz-Signature=9620505493b7e78166ac812f7aed14b75927d4fd40aa0648ab7f08a1489071a9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream'
*   Trying 185.199.108.133:443...
* Connected to objects.githubusercontent.com (185.199.108.133) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[...hangs...]

Calling the first URL results in a 302 redirect to a different url, which is duly followed by curl (due to -L) - but, on attempting to negotiate TLS for the second location, the remote server does not respond.

Comparing this with a similar call from the VM host machine (a Mac):

$ curl -V
curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

$ curl -vvv -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
*   Trying 192.30.255.112:443...
* Connected to github.com (192.30.255.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
[...]
< HTTP/2 302
[...]
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T223206Z&X-Amz-Expires=300&X-Amz-Signature=cf2a257721dc3cd0955a4e32c5b58074340ad6c305101d1338e5f186d64c68a2&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream'
*   Trying 185.199.111.133:443...
* Connected to objects.githubusercontent.com (185.199.111.133) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
[...]
< HTTP/2 200
[expected output]
* Connection #1 to host objects.githubusercontent.com left intact

A similar call from a Debian-based Raspberry Pi on the same network gave a similar successful output (one notable change - * (304) (OUT/IN), TLS handshake, ...) lines instead had TLS v1.[0-3], as in the unsuccessful case)

Of note when comparing these two:

  • The sequence of TLS negotiation steps for the first request differ between the successful and unsuccessful case - the successful one skips some Certificate Status and Supplemental data steps
  • The IP addresses for objects.githubuserconcent.com differ - but nslookup objects.githubusercontent.com indicates that 185.199.[108-111].133 are all expected IPs

Debugging and investigation

Things I've tried/investigated to resolve this issue:

  • I did consider downgrading the version of curl on the VM to match that on the host machine (and the Debian Pi), but apt-cache madison curl didn't show any older versions, and in any case I saw several warnings against downgrading curl on other questions/forums
  • I double-checked that the DigiCert Root CA is present in /etc/ssl/certs
  • I used openssl s_client to check that my VM can connect to the target domain:
$ openssl s_client -connect objects.githubusercontent.com:443
CONNECTED(00000003)
  • I tried directly curling the objects.githubusercontent.com url (with and without -k) - hangs at the same place in the TLS negotiation
  • I tried explicitly setting the set of allowed ciphers (curl -vvv -sfL --ciphers "AES128-SHA256:AES256-SHA384" https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt) - same behaviour
  • I did consider that the firewall on my OPNSense router might be blocking the incoming part of the TLS negotiation - however, I suspect that's unlikely because:
    • I viewed the "live logs" of the firewall, and there were no blocked requests from 185.199.* while testing
    • Directly curling the objects.githubusercontent.com url fails, so there's nothing weird going on related to the redirection
    • I doubt that there's an issue with a firewall in Multipass itself, otherwise the linked articles would have mentioned it
  • I tried using wget instead, which appeared to fail in the same place:
$ wget https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
--2022-11-28 15:11:34--  https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
Resolving github.com (github.com)... 192.30.255.112
Connecting to github.com (github.com)|192.30.255.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T231137Z&X-Amz-Expires=300&X-Amz-Signature=4e96a7bf7f4ed5be5caa64f54fa653d584b089d15226cfa33131897eff9e7c39&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream [following]
--2022-11-28 15:11:38--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T231137Z&X-Amz-Expires=300&X-Amz-Signature=4e96a7bf7f4ed5be5caa64f54fa653d584b089d15226cfa33131897eff9e7c39&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
[...hangs...]
  • I note that there are some tools known to interfere with Multipass on Mac, including OpenVPN (which I'm using), but I don't think this is the case here, as DNS resolution and ping work as expected. I'm pretty uncertain about my reading of this page, though, so feel free to correct me.

EDIT: I found this similar issue - I'm working through the steps there.

414
1 + 1
ssl
tls
ke flag
scubbo
Note this was also asked on the [main SO site](https://stackoverflow.com/questions/74607542/tls-negotiation-fails-from-multipass-vm-but-succeeds-from-host-machine-or-other).
0
Reply
Score:0
avatar Ubuntu
ke flag
scubbo

Turning off my OpenVPN resolved this issue. I'll keep this question open until I find a way around that, though, as that's not a very satisfactory solution.

EDIT: as @Steffan pointed out, this was due to an MTU mismatch - my Multipass interface had (standard) MTU=1500, but my VPN was using 1420. I set up a second VPN to my home network, used that to reconfigure my first VPN to MTU=1500, and now everything works as expected (on my first VPN).

Steffan, if you want to submit this as an answer, I'd be happy to accept it to give you the points.

+ 2
Steffen Ullrich avatar
in flag
Steffen Ullrich
If it works without VPN but "hangs" on some data with enabled VPN it is usually a MTU problem. The part about MTU adjustments in https://hamy.io/post/0003/optimizing-openvpn-throughput/ might help.
0
Reply
ke flag
scubbo
Thanks! As per [here](https://github.com/canonical/multipass/issues/495#issuecomment-1330199153), I tried changing the MTU of the Multipass interface to match the VPN, but was unable to do so. I'm currently away from home and so don't want to risk making changes to the VPN that might leave it inoperable.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.