I am facing random occurrences of 502 Proxy Error. We have a 3rd party application running in our server. It uses a reverse proxy to serve the request. The Proxy server is returning 502 Proxy Errors in a random manner. A similar request that failed 5 mins back passes in future tries. The number of occurrences is quite high
I am running out of ideas to debug. I am facing this in production setup. In dev setup which is exactly the same except for the load in the application I am unable to reproduce the issue.
Resource utilization for the server is not even reaching 50% in production.
ProxyTimeout 120
# Required apache modules: proxy_module, proxy_http_module and proxy_wstunnel_module
<Location />
# Pass the HTTP protocol request header to the backend server if SSL is inactive.
RequestHeader set "X-Forwarded-Proto" "http" env=!HTTPS
ProxyPass http://localhost:8080/
# Prevent apache incompatibility with web service clients
# that send a "100 Continue" header.
# Variant for apache 2.4.40 and later.
<IfVersion >= 2.4.40>
Proxy100Continue Off
</IfVersion>
</Location>
Client code that call this proxy server is set a timeout of 60 sec. In the access log, I see the HTTP request took around 0-12 seconds before 502 Proxy Error was thrown. (Last field is time to serve in sec %T)
44.193.158.108 - - [08/Mar/2023:06:03:44 +0000] "POST /otrs/nph-genericinterface.pl/Webservice/ScannerWebservice/TicketSearch?UserLogin=xxx&Password=xxxxx HTTP/1.1" 502 520 8
44.193.158.108 - - [08/Mar/2023:06:09:14 +0000] "POST /otrs/nph-genericinterface.pl/Webservice/ScannerWebservice/TicketSearch?UserLogin=xxx&Password=xxxxx HTTP/1.1" 502 520 0
3.209.245.186 - - [08/Mar/2023:06:09:13 +0000] "POST /otrs/nph-genericinterface.pl/Webservice/ScannerWebservice/TicketDetails?UserLogin=xxx&Password=xxxxx HTTP/1.1" 502 522 1
In the application log, the following is returned
[Wed Feb 22 12:10:13.688023 2023] [proxy_http:error] [pid 9575] (104)Connection reset by peer: [client 8.29.230.55:21979] AH01102: error reading status line from remote server localhost:8080
[Wed Feb 22 12:10:13.688122 2023] [proxy:error] [pid 9575] [client 8.29.230.55:21979] AH00898: Error reading from remote server returned by /otrs/nph-genericinterface.pl/Webservice/ScannerWebservice/TicketDetails
