Score:0

GCP External HTTPs Load Balancer - 404 - 503 - SSL Exception (Remote host terminated connection, read handshake, socket closed & upstream connect)

in flag

We're load testing a MIG (with 2 instances) hosted behind the HTTPs load balancer using JMeter.

Observation 1: We randomly receive 404 error and 503 error, for 404 we see an entry get created within load balancer monitoring NO_BACKEND_SELECTED (other than our actual MIG backend). Further, for 503 we see an entry get created within load balancer monitoring FRONTEND_5XX.

Based on GCP:

NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected. FRONTEND_5XX - An internal error occurred before the GFE could select a backend. The GFE returned 5XX to the client.

The above statement doesn't assist with respect to troubleshooting or getting the same resolved or isolation of the cause for the issue. we didn't find anything w.r.t. these error messages within GCP docs or other articles.

Observation 2: We randomly receive random SSL exceptions; Remote host terminated connection, read handshake, socket closed & upstream connect @ JMeter's end.

Steps taken

  1. Changing Keep Alive on the backend servers to 620 sec (GFE has Keep Alive of 600 secs)
  2. Created custom SSL policy (minimum tls set to 1.1)
  3. Increased the backend timeout from default 30 to 65 seconds

So, we are looking @ what are we missing or what else can we fine-tune/modify for testing purposes in order to get the above mentioned issues resolved.

Thank you. Gaurav_N17

Score:0
st flag

It sounds like sometimes might have no healthy instances on the backend.

I suggest you to Monitor autoscaling charts and logs and take a look at Commonly returned status message.

These steps are supposed to give you a clue about what's going on on backend and load balancer

in flag
we've already analyzed that during load testing and there's no issue with instances running at the backend. Any idea w.r.t. NO_BACKEND_SELECTED and FRONTEND_5XX error messages.
st flag
@Gaurav_N17, I'm wondering if backend instances might have gotten overloaded at load testing time, and got unable to answer request due to high load (CPU, I/O, processing, max HTTP open connections, etc). It's not a application issue, but a max resources limitation
in flag
yes looks like the same, we observed high load within the database servers during the load test. We'll be increasing the resources and then carry out another set of load testing.
Score:0
in flag

after load testing, we concluded that it was at the high resource utilization at the database end which was causing the issue. we vertically scaled the DB resources. thank you.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.