Score:1

Application works locally, but fails with 502 in cloud run - "upstream connect error or disconnect/reset before headers. reset reason: protocol error"

cn flag

I have an application built as a Docker image that works correctly locally, but fails in Google Cloud Run with "upstream connect error or disconnect/reset before headers. reset reason: protocol error"

The application is listening on 0.0.0.0:8080 and does not require authentication. I'm running it in Docker locally with

docker run --rm --name myapp -p 8080:8080 myapp:1.0

With curl locally, I get a successful response on /

$ curl -v localhost:8080
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Origin
< Date: Sat, 24 Jul 2021 13:54:11 GMT
< Content-Length: 2
<
* Connection #0 to host localhost left intact
OK* Closing connection 0

I've deployed this to Google cloud run as a service, and when making the first request, I see log output from my application in the cloud run logs showing that it has started and is listening on port 8080.

However, a request fails with

"upstream connect error or disconnect/reset before headers. reset reason: protocol error"

Full output from curl -v https://cloudrun-srv-stuff.a.run.app/ is as follows

*   Trying 216.239.36.53...
* TCP_NODELAY set
* Connected to cloudrun-srv-lz5wvvl32a-ez.a.run.app (216.239.36.53) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.a.run.app
*  start date: Jun 28 01:08:46 2021 GMT
*  expire date: Sep 20 01:08:45 2021 GMT
*  subjectAltName: host "cloudrun-srv-stuff.a.run.app" matched cert's "*.a.run.app"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f83b4810a00)
> GET / HTTP/2
> Host: cloudrun-srv-stuff.a.run.app
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 502
< content-length: 87
< content-type: text/plain
< date: Sat, 24 Jul 2021 13:58:13 GMT
< server: Google Frontend
< x-cloud-trace-context: 1370fab1a02839b367048f855593cbef/5890123523738625787;o=1
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<

What are the next steps in debugging this?

John Hanley avatar
cn flag
1) Do you have Cloud Run **end-to-end** http2 enabled? 2) Show the gcloud deploy command. 3) Show the code that sets up the application listener. 4) Check the Cloud Run logs for this service. 5) Add the curl command-line option **--http1.1** and test again.
cn flag
John's answer above was the issue - http2 end to end was enabled, but my Go code does not support H2C.
Score:5
cn flag

John Hanely's response to my question identified the issue. https://cloud.google.com/run/docs/configuring/http2#console allowed me to disable http/2 end-to-end, and this resolved the issue.

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.