Half of data lost between HAProxy and backend server

I have a setup like this on Ubuntu

Browser --> HAProxy --> Backend server

The backend server is an ASP.NET Core web application.

It works 99.9% of the time except when a binary file is being uploaded (simple POST with multipart form-data), in which case I get the error:

System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component. 

   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)

   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)

   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)

   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)

   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenStore.GetRequestTokensAsync(HttpContext httpContext)

   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.ValidateRequestAsync(HttpContext httpContext)

   at Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.ValidateAntiforgeryTokenAuthorizationFilter.OnAuthorizationAsync(AuthorizationFilterContext context)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)

   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)

   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

So, I have been trying to investigate why.

I tried the ASP.NET Core backend directly, it works.

So I logged the incoming data to the backend and found that if I do it directly then I get the expected request headers and then the uploaded file in the body.

Then I logged the incoming data to the backend via HAProxy and half of the data is missing.

Ie. the first data seen by the backend appears to be

04 E2 22 FC 60 FF 2B E1  BF 85 D2 75 F9 44 94 86

but these are the bytes roughly halfway through my file, which is 25,126 bytes. I don't see any header info at all.

I am prepared to accept the my logging is imperfect and not necessarily accurate. It seems like somehow the request was split in two, or a buffer has been filled and then written over.

What could the problem be?

global

    chroot /var/lib/haproxy
    log /dev/log    local0 info

    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group users
    daemon
        
    lua-load /home/user/haproxy-mapping.lua

    ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS:!AESCCM
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    tune.ssl.default-dh-param 2048


defaults
    log     global
    mode    http

    timeout connect 30000
    timeout client  50000
    timeout server  50000
    option forwardfor
    option http-server-close #have also try http-keep-alive


frontend httpfront
    mode http
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }


frontend web_front_end

    bind *:443 ssl crt /home/.....file.pem
    mode http

     log /var/lib/haproxy/dev/log local0 info

        # Rate limiting
        stick-table  type ip  size 100k  expire 600s  store http_req_rate(60s) #store up to 100k requests for 60s, see if over 60s there are more than 600
        http-request track-sc0 src
        http-request deny deny_status 429 if { sc_http_req_rate(0) gt 600 }


    # Ensure we have a clean state to start with
    http-request del-header X-SERVER-SNI

    # Set the concatenated value of the SNI value to a temporary header
    http-request set-header X-SERVER-SNI haproxy.%[ssl_fc_sni] if { ssl_fc_sni -m found }

    # Set the value of the header to a transaction-level variable
    http-request set-var(txn.fc_sni) ssl_fc_sni #hdr(X-SERVER-SNI) if { hdr(X-SERVER-SNI) -m found }

    #use Lua code to determine which backend to send to
    use_backend %[lua.legacy_new_backend]





backend backendnodes_1_https
    balance roundrobin
    option forwardfor
    server node1 127.0.0.1:446 ssl verify none sni var(txn.fc_sni)