GCP External HTTPs Load Balancer - 404 - 503 - SSL Exception (Remote host terminated connection, read handshake, socket closed & upstream connect)

We're load testing a MIG (with 2 instances) hosted behind the HTTPs load balancer using JMeter.

Observation 1: We randomly receive 404 error and 503 error, for 404 we see an entry get created within load balancer monitoring NO_BACKEND_SELECTED (other than our actual MIG backend). Further, for 503 we see an entry get created within load balancer monitoring FRONTEND_5XX.

Based on GCP:

NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected. FRONTEND_5XX - An internal error occurred before the GFE could select a backend. The GFE returned 5XX to the client.

The above statement doesn't assist with respect to troubleshooting or getting the same resolved or isolation of the cause for the issue. we didn't find anything w.r.t. these error messages within GCP docs or other articles.

Observation 2: We randomly receive random SSL exceptions; Remote host terminated connection, read handshake, socket closed & upstream connect @ JMeter's end.

Steps taken

1. Changing Keep Alive on the backend servers to 620 sec (GFE has Keep Alive of 600 secs)
2. Created custom SSL policy (minimum tls set to 1.1)
3. Increased the backend timeout from default 30 to 65 seconds

So, we are looking @ what are we missing or what else can we fine-tune/modify for testing purposes in order to get the above mentioned issues resolved.

Thank you. Gaurav_N17

It sounds like sometimes might have no healthy instances on the backend.

I suggest you to Monitor autoscaling charts and logs and take a look at Commonly returned status message.

These steps are supposed to give you a clue about what's going on on backend and load balancer

after load testing, we concluded that it was at the high resource utilization at the database end which was causing the issue. we vertically scaled the DB resources. thank you.