Summary
Running curl -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
from a Multipass VM appears to result in a hung TLS negotiation, as there is no (IN), TLS handshake, Server hello (2)
response for the 302-redirected URL.
Detail
I'm trying to install k3s on a Multipass VM on my Mac, as documented here and here. The k3s installation process involves downloading and running an installation script (curl -sfL https://get.k3s.io | sh -
) - however, this script hangs partway through when run on the Multipass VM:
[INFO] Finding release for channel stable
[INFO] Using v1.25.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
[...hangs...]
From stepping through the script, I found that it's hanging on a call to curl -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
. I tried to replicate this call directly (with verbose flags set) - selected output below:
$ curl -V
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd
$ curl -vvv -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
* Trying 192.30.255.112:443...
* Connected to github.com (192.30.255.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
[...]
< HTTP/2 302
[...]
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T222055Z&X-Amz-Expires=300&X-Amz-Signature=9620505493b7e78166ac812f7aed14b75927d4fd40aa0648ab7f08a1489071a9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream'
* Trying 185.199.108.133:443...
* Connected to objects.githubusercontent.com (185.199.108.133) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[...hangs...]
Calling the first URL results in a 302 redirect to a different url, which is duly followed by curl (due to -L
) - but, on attempting to negotiate TLS for the second location, the remote server does not respond.
Comparing this with a similar call from the VM host machine (a Mac):
$ curl -V
curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets
$ curl -vvv -sfL https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
* Trying 192.30.255.112:443...
* Connected to github.com (192.30.255.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
[...]
< HTTP/2 302
[...]
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T223206Z&X-Amz-Expires=300&X-Amz-Signature=cf2a257721dc3cd0955a4e32c5b58074340ad6c305101d1338e5f186d64c68a2&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream'
* Trying 185.199.111.133:443...
* Connected to objects.githubusercontent.com (185.199.111.133) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
[...]
< HTTP/2 200
[expected output]
* Connection #1 to host objects.githubusercontent.com left intact
A similar call from a Debian-based Raspberry Pi on the same network gave a similar successful output (one notable change - * (304) (OUT/IN), TLS handshake, ...)
lines instead had TLS v1.[0-3]
, as in the unsuccessful case)
Of note when comparing these two:
- The sequence of TLS negotiation steps for the first request differ between the successful and unsuccessful case - the successful one skips some
Certificate Status
and Supplemental data
steps
- The IP addresses for
objects.githubuserconcent.com
differ - but nslookup objects.githubusercontent.com
indicates that 185.199.[108-111].133
are all expected IPs
Debugging and investigation
Things I've tried/investigated to resolve this issue:
- I did consider downgrading the version of
curl
on the VM to match that on the host machine (and the Debian Pi), but apt-cache madison curl
didn't show any older versions, and in any case I saw several warnings against downgrading curl on other questions/forums
- I double-checked that the DigiCert Root CA is present in
/etc/ssl/certs
- I used
openssl s_client
to check that my VM can connect to the target domain:
$ openssl s_client -connect objects.githubusercontent.com:443
CONNECTED(00000003)
- I tried directly curling the
objects.githubusercontent.com
url (with and without -k
) - hangs at the same place in the TLS negotiation
- I tried explicitly setting the set of allowed ciphers (
curl -vvv -sfL --ciphers "AES128-SHA256:AES256-SHA384" https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
) - same behaviour
- I did consider that the firewall on my OPNSense router might be blocking the incoming part of the TLS negotiation - however, I suspect that's unlikely because:
- I viewed the "live logs" of the firewall, and there were no blocked requests from 185.199.* while testing
- Directly curling the
objects.githubusercontent.com
url fails, so there's nothing weird going on related to the redirection
- I doubt that there's an issue with a firewall in Multipass itself, otherwise the linked articles would have mentioned it
- I tried using
wget
instead, which appeared to fail in the same place:
$ wget https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
--2022-11-28 15:11:34-- https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
Resolving github.com (github.com)... 192.30.255.112
Connecting to github.com (github.com)|192.30.255.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T231137Z&X-Amz-Expires=300&X-Amz-Signature=4e96a7bf7f4ed5be5caa64f54fa653d584b089d15226cfa33131897eff9e7c39&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream [following]
--2022-11-28 15:11:38-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/546bdb42-d049-41b6-89f8-6595cecb44cf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221128T231137Z&X-Amz-Expires=300&X-Amz-Signature=4e96a7bf7f4ed5be5caa64f54fa653d584b089d15226cfa33131897eff9e7c39&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dsha256sum-amd64.txt&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
[...hangs...]
- I note that there are some tools known to interfere with Multipass on Mac, including OpenVPN (which I'm using), but I don't think this is the case here, as DNS resolution and ping work as expected. I'm pretty uncertain about my reading of this page, though, so feel free to correct me.
EDIT: I found this similar issue - I'm working through the steps there.