Join Log in
Score:0
avatar Server

HAProxy with same request sometimes responding with NOSRV while backend is up

tr flag
SRedouane

Im making the same request(POST /user/oauth) to haproxy but sometimes i get 503, sending requests directly to the backend always works

here is the logs:

  1. local_server~ app/s2 0/0/0/100/100 200 813 - - --VN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
  2. local_server~ app/s1 0/0/1/107/108 200 820 - - --NI 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
  3. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 - - SCNN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
  4. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 - - SCNN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
  5. local_server~ app/s2 0/0/1/106/107 200 831 - - --NI 2/2/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
  6. local_server~ app/s1 0/0/1/3/6206 101 337 - - --NI 3/3/1/1/0 0/0 "GET / HTTP/1.1"
  7. local_server~ app/s1 0/0/1/83/84 200 258 - - --VN 2/2/0/0/0 0/0 "POST /transaction/concludeTransaction HTTP/1.1"
  8. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 - - SCNN 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"
  9. local_server~ app/s2 0/0/1/107/108 200 831 - - --NI 1/1/0/0/0 0/0 "POST /user/oauth HTTP/1.1"

as you can see the POST /user/oauth sometimes go to s1 sometimes go to s2 and sometimes to NOSRV.

sometimes it pass 10 times in a row and sometimes it fails 10 times in a row.

here is my config:

defaults
log     global
mode    http
option  httplog
option  dontlognull
timeout connect 10000
timeout client  50000
timeout server  50000
retries 3
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend local_server
bind *:8080 ssl crt /etc/letsencrypt/live/xxx/sslcert.pem
mode http
default_backend app

backend app
balance roundrobin
cookie SERVERID insert indirect nocache
http-send-name-header Host
server s1 172.26.13.134:3000 check cookie s1
server s2 172.26.13.134:3000 check cookie s2
http-request add-header X-Server %b

Edit:

i did redirect the request the a quick python server running on local host and still the problem persist. so i did edit haproxy config file to have only server s1 127.0.0.1:8086 check cookie s1

and run the python script: python3 -m http.server 8086 --bind 127.0.0.1&

when i hit from postman i got like 3 succ on a row than a 503.

107
0 + 0
richv avatar
in flag
richv
might be a good idea to add health check to your nodes https://www.haproxy.com/blog/how-to-enable-health-checks-in-haproxy/ ... you can compare those NOSERV timestamps to node down time and see if both nodes are down at a time when you're getting a NOSERV response
0
Reply
tr flag
SRedouane
it is already there "check", also calling directly the node always works and the nodes are never down. when i make an api call from postman i get these random 503
0
Reply
Score:0
avatar Server
tr flag
SRedouane

I believe the problem is within the version, i read somewhere that version 1.9 work better than 2.* i downloaded the source code and compiled with the following options make TARGET=linux-glibc USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_REGPARAM=1 USE_SYSTEMD=1 USE_THREAD=1

as you can see I omitted on purpose the lua5 option which is the only difference with the one you get from apt-get install.

the resulted binay work very well without any issue

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.