Join Log in
Score:0
avatar Server

Apache 2.4 MPM event module causing intermittent 502 errors and slow response times behind an AWS Application Load Balancer (ALB)

cn flag
danw

tldr; How do you configure the Apache MPM event module as a functional reverse proxy when running behind a high throughput AWS Application Load Balancer?

Layout

AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers

Using the configuration setting shown below the Apache MPM event module is not stable behind an AWS app load balancer. Response times are greatly increased and requests intermittently fail for 502 errors returned by the ALB.

Worker configuration

ServerLimit 250
StartServers 100
MinSpareThreads 75
MaxSpareThreads 250
ThreadLimit 64
ThreadsPerChild 64
MaxRequestWorkers 8000

ProxyPass config

enablereuse=on acquire=10 connectiontimeout=2 timeout=55 max=400 retry=0 keepalive=on

MPM event vs MPM worker

Switching to the MPM worker module shows instant improvements in response times and the 502 errors disappear completely.

Target Response Times - MPM event to MPM worker

502 Errors - MPM event to MPM worker

Debugging

I managed to trace the 502 errors returned by the ALB to 408 errors returned by Jetty. Jetty seems to be throwing the errors because of timeouts reading the request.

Running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8.

Interesting notes

  • Requests though an AWS Network Load Balancer (NLB) to the same Apache instances respond quickly and don't show any 502 errors
242
0 + 0
cn flag
shearn89
The difference with the NLB may be because NLBs work at a lower layer than ALBs. Can I ask why apache is in the middle? Why not just have the ALB route to the Jetty servers?
0
Reply
cn flag
danw
@shearn89 its a WAF: https://github.com/SpiderLabs/ModSecurity
0
Reply
cn flag
shearn89
Why not use AWS WAF instead, which is probably cheaper than running a whole EC2 instance? plus it's managed for you, so you just have to enable some AWS-managed rules and you're good to go!
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.