HAProxy health-check fails with Layer7 wrong status, code: 400, info: "HTTP status check returned code 400

I have an HAProxy server that I'm using as a L7 load balancer for my k8s nodes. My cluster is istio enabled and I have an istio-ingressgateway service exposed via NodePort

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                      AGE
istio-ingressgateway   NodePort    10.11.140.167   <none>        15021:30301/TCP,80:31916/TCP,443:31517/TCP,15012:30768/TCP,15443:32020/TCP   11d

From the HAProxy server I'm trying to perform health-checks on /healthz/ready endpoint. I'm using HAProxy 1.8 and my haproxy.cfg is as follows:

global
    log         /dev/log    local0
    log         /dev/log    local1 notice
    chroot      /var/lib/haproxy
    pidfile     /var/run/rh-haproxy18-haproxy.pid

    user        haproxy
    group       haproxy
    daemon
    stats       socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

    spread-checks  21

    # Default SSL material locations
    ca-base     /etc/ssl/certs
    crt-base    /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    # An alternative list with additional directives can be obtained from
    #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  redispatch
    retries                 3

    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 10000
    balance                 roundrobin


frontend http-80
        bind *:80
        mode http
        option httplog
        default_backend www-80

backend www-80
        balance roundrobin
        mode http
        option httpchk /healthz/ready HTTP/1.1
        http-check expect status 200
        server backendnode1 155.13.200.29:31916 check port 30301 fall 3 rise 2 inter 1597
        server backendnode2 155.13.200.28:31916 check port 30301 fall 3 rise 2 inter 1597
        server backendnode3 155.13.200.27:31916 check port 30301 fall 3 rise 2 inter 1597


frontend health-80
    bind *:8080
    acl backend_dead nbsrv(www-80) lt 1
    monitor-uri /haproxy_status
    monitor fail if backend_dead

listen stats # Define a listen section called "stats"
    bind :9000 # Listen on localhost:9000
    mode http
    stats enable  # Enable stats page
    stats hide-version  # Hide HAProxy version
    stats realm Haproxy\ Statistics  # Title text for popup window
    stats uri /haproxy_stats  # Stats URI
    stats auth haproxy:passwd  

I am using HTTP/1.1 for backend health-check because istio-ingressgateway is not accepting HTTP/1.0 requests, it results in error code 426.

Hitting the backend from HAProxy server results in success:

curl -I  http://155.13.200.29:31916/healthz/ready
HTTP/1.1 200 OK
date: Fri, 11 Jun 2021 07:21:09 GMT
x-envoy-upstream-service-time: 0
server: envoy
transfer-encoding: chunked

However, the HAProxy health-check still won't pass. I'm getting the following errors:

Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode2 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 2ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode2 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 2ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[11795]: [WARNING] 161/071821 (11795) : Former worker 11798 exited with code 0
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode3 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode3 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

I understand that status code 400 occurs for bad requests. Is there something miss configured in my haproxy.cfg? I feel like it's the way I am trying to send HTTP/1.1 request for health-check. But, I'm not sure what else to add or what to modify in the configuration.

Check the manual, you can't specify version without specifying the method:

option httpchk <method> <uri> <version>

in your case, I'd try

option httpchk GET /healthz/ready HTTP/1.1

Also: learn about tcpdump, very interesting to watch the communication between systems and find out what's wrong.