Score:0

gRPC via CloudFlare results in HTTP/2 internal error code 2

ar flag

My setup:

.Net gRPC Server <-> Nginx <-> CloudFlare <-> gRPC client (C#/Python)

My .Net gRPC Server configured to support insecured http2, listen at port 50052:

webBuilder.UseStartup<StartupGrpc>().UseUrls($"http://*:50052");
webBuilder.ConfigureKestrel(serverOptions => { serverOptions.ConfigureEndpointDefaults(listenOptions => { listenOptions.Protocols = HttpProtocols.Http2; }); });

Nginx is set to grpc_pass as follow:

server {
    server_name grpc.mydomain.com;
    listen      443 ssl http2;
    ssl_certificate    /etc/nginx/cf_origin_ssl/mydomain.pem;
    ssl_certificate_key /etc/nginx/cf_origin_ssl/mydomain.key;

    proxy_cache off;
 
    location / {
        grpc_pass grpc://localhost:50052;   
    }
}

server {
    server_name mydomain.com;
    listen      443 ssl http2;
    ssl_certificate    /etc/nginx/cf_origin_ssl/mydomain.pem;
    ssl_certificate_key /etc/nginx/cf_origin_ssl/mydomain.key;

    proxy_cache off;
 
    location / {
        proxy_pass localhost:50051;   
    }
}

CloudFlare: Network/gRPC -> On, SSL/TLS -> Full(strict) (with Origin Certificates generated by CloudFlare). I tested, and my web server at mydomain.com worked fine. However, gRPC calls from .Net/C# gRPC Client returns:

Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. IOException: The request was aborted. Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' (0x2).", DebugException="System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'INTERNAL_ERROR' (0x2).
    --- End of inner exception stack trace ---
    at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
    at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
    at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
    at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
    at Grpc.Net.Client.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
    at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")

I also tried to make gRPC calls from Python, and got a similar error:

Traceback (most recent call last):
    ...
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses"
    debug_error_string = "{"created":"@1634609018.116476058","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3158,"referenced_errors":[{"created":"@1634609018.116472621","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":147,"grpc_status":14}]}"
    >

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
    ...
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
    File "/home/user/miniconda/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.INTERNAL
    details = "Received RST_STREAM with error code 2"
    debug_error_string = "{"created":"@1634609018.553728473","description":"Error received from peer ipv4:172.67.179.119:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"Received RST_STREAM with error code 2","grpc_status":13}"
    >

In both cases, the gRPC requests did get through CloudFlare, Nginx and reach my gRPC server (The remote procedures got executed). Nginx logs also reported with 200 success code:

116.110.42.123 - - [19/Oct/2021:01:25:22 +0000] "POST /greet.Greeter/CsharpSayHello HTTP/2.0" 200 64 "-" "grpc-dotnet/2.40.0.0" "116.110.42.123" "grpc.mydomain.com" sn="grpc.mydomain.com" rt=0.002 ua="127.0.0.1:50052" us="200" ut="0.000" ul="71" cs=-
116.110.42.123 - - [19/Oct/2021:01:27:57 +0000] "POST /greet.Greeter/CsharpSayHello HTTP/2.0" 200 68 "-" "grpc-python/1.41.0 grpc-c/19.0.0 (linux; chttp2)" "116.110.42.123" "grpc.mydomain.com" sn="grpc.mydomain.com" rt=0.001 ua="127.0.0.1:50052" us="200" ut="0.000" ul="75" cs=-

I googled a lot about CloudFlare gGRPC and Nginx, but couldn't figure out what is wrong.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.