Score:1

Why is Nginx truncating the JSON streaming response?

in flag

Our stack is Client(Browser) <-> Nginx Reverse Proxy <-> Webserver(Flask+Gunicorn) <-> Golang gRPC server

The problem is when the client makes a call to the /realtimedata endpoint, Flask then opens the gRPC connection and starts receiving data via a server->client unidirectional stream. It then passes it back to the client. When I run this without Nginx, I get all responses. When running with Nginx, some responses get truncated. For example, if we expect:

{
    "source": "serviceA",
    "timestamp": 123456789,
    "data": {
        "1": 24.55667,
        "2": -456.5656,
        ...
        "200": 5.678
    }
}

We get

{
    "source": "serviceA",
    "time

Then

        stamp": 123456789,
    "data": {
        "1": 24.55667,
        "2": -456.5656,
        ...
        "200": 5.678
    }
}

This would be printed in console.log. I have proxy_buffering off; in the nginx configuration otherwise no data makes it to the browser. Not sure how to resolve this issue.

Here is a minimum, reproducible example.

UPDATE: I've ran the minimum reproducible example with Apache2 instead of Nginx and experiencing the same random truncations.

cn flag
What do you mean by "We get [...] and then [...]"?! That you receive all the data anyway? I don't see why that would be an issue.
Paul Côté avatar
in flag
I print the response in the console on the browser. So I'll receive for example one half of the response, then the other. But it is an issue because `response.JSON()` yields an error when it's truncated like that.
Score:0
ve flag

I had the similar behavior from Apache(!) breaking up a html file respone from django. It always broke at the same position without any reason. I finally found that I accidentally did install a mod_wsgi package into Apache from a Python version that did not match the version Apache was calling my app in. Maybe it helps you.

Paul Côté avatar
in flag
Hey thanks I'll check that out. I ended up using websockets to solve the issue. The truncating stopped.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.