Score:0

HAProxy Timeouts and Streaming Backend - Hangs Forever

ru flag
Sam

I have a backend behind haproxy that streams a file to the client using a CGI script. I'm trying to set a timeout for my backend servers. All works fine, except for this timeout.

When I set the "timeout server" option in the haproxy config to 1 second (just to test - the backend takes 2-3 seconds at a minimum), and make a request, after 1 second it puts an entry into the log file for the backend request (before the request is completed), and from there, it hangs forever. Client never receives a response, no error, nothing.

I've looked through all of the docs for haproxy config and I can not find any timeout option that makes sense here. But this seems like very strange default behavior, to hang forever if the timeout is reached without a complete response (I've since realized setting is supposed to be a timeout just for the headers - but I can't seem to find another timeout setting for the backend).

Does haproxy just not support any kind of streaming HTTP response? I understand that we can't send a header - but can I set it to just send whatever data it has so that the client picks up that it is malformed?

Here is a log line for one of these requests that "timed out":

Apr 28 09:32:51 print haproxy[29896]: 127.0.0.1:37662 [28/Apr/2022:09:32:47.317] http_front http_back/SERVERNAME 2/3007/23/46/4080 200 150 - - sD-- 1/1/0/0/+3 0/0 "POST / HTTP/1.1"

Full haproxy config:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option redispatch
    retries 3
    timeout queue 15m # How long a client should wait for a server to become available.
    #maxconn 10000
    timeout client 15m
    timeout connect 1s
    timeout server 10s
    rate-limit sessions 10000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend http_front
    bind *:88
    stats uri /haproxy?stats
    default_backend http_back
    #maxconn 10000

backend http_back
    balance leastconn
    retry-on all-retryable-errors
    default-server maxconn 1 observe layer7 error-limit 1 on-error mark-down
    server NAME IP:PORT

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.