AWS ALB Healthchecks Against ECS Services Periodically Failure For No Discernable Reason

Dustin Oprea

7/19/24, 12:46 PM

We completely host a constellation of services in AWS (no external dependencies as far as these services go). We periodically receive healthcheck failures (502 as public services try to contact the internal service ALBs), as frequent as every hour or two. The services experience no disruption whatsoever.

I've tried all manner of healthcheck settings (long and short durations, high and low counts [until considered successful or failed]). When I've looked at the HTTP log in the past, I believe that there were no records whatsoever for the failed requests; I'd just assumed that the service was downed before the request finished and one could be written. We have regular activity, but it wouldn't be considered high-volume. Congestion is not a factor (which might disrupt normal requests, but isn't per the above).

We have more than one load-balanced instance per service.

This is a long-running issue and I've periodically searched and tried whatever reasonable approaches have been suggested, but I've had no luck learning anything further.

The platform is largely uWSGI (Python) behind Nginx.

How could I further debug this?

1 + 0

amazon-web-services

amazon-ecs

amazon-alb

Score:1

Server

t3ln3t

7/20/24, 2:03 AM

This is a super PITA to troubleshoot with what you can get from the console. If you really and truly want to dig into this, pony up whatever they want these days for developer support. It used to be $20/month ... but you will get access to premium support folks. They're smart and paid to dig into this sort of thing! You mentioned that you have tried all manner of settings... the service quotas in here are a mess of stuff! You're likely just tripping a simple service quota/limit that's placed on all cloud service accounts from the get go? If you pony up for developer support, even just for a month (or as needed) you can get pretty darn quick response from their premium support folks who can put in quota increases. There are some limitations still, but those application service limits are probably what you're running into. Start here: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html

If you can adjust your solution to fit within the confines of the default limits, that's probably best, overall. If you do get a quota increase from the support folk, please document it?! Someone doing this later might not be you and they will get tripped up on this too. Do them a solid?

I hope this helps? Good luck!

+ 3

Dustin Oprea

7/22/24, 4:26 AM

None of those limits can explain the heartbeat aperiodically failing even while the system is still operational in general. Unfortunately, the developer plan is $29 *OR* 3% of the month's bill, whichever is bigger, and that cost significantly exceeds the value of the fix where the problem doesn't actually disrupt operations.

t3ln3t

8/4/24, 1:48 AM

Have you tried using the AWS CLI to extract further information at all? If you throw the verbose switches it can expose more information that could prove useful? I get the billshock thing. It's how they getcha! in the end!!!

Dustin Oprea

8/14/24, 8:42 AM

Use the AWS CLI to look at what? Verbose switches to what? This is an ECS deployment. We tell it to use a new task-definition, it starts a number of new instances, and the TG reports when they go healthy. There's nothing to request verbosity from (aside from configuring access-logs in the ALB, but this isn't a "command" thing). This is a network-level phenomenon. Its lower-level details are somewhat obscured from our view.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: AWS ALB Healthchecks Against ECS Services Periodically Failure For No Discernable Reason

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.