I upgraded Ubuntu Server from 22.10 to 23.04 that hosts some websites over Apache2 as well as a few .NET websites which was working normally but following the upgrade, requests for all the Apache2 websites just timeout where a WordPress one shows a 504 gateway timeout error and the .NET ones do load but then after a little while also then stop responding.
Restarting Apache2 has no effect on the Apache2 websites but does make the Net ones work again briefly.
I set up a new blank 23.04 Ubuntu instance (downloading the iso from the Ubuntu website), installed Apache2, MariaDB and .NET 7 and moved everything across but after configuring Apache it is behaving exactly the same as the old server so it must be related to the way it is configured although this has not changed for over a year with only updates being applied.
The old server is hosted on DigitalOcean as a VPS whereas the new server is hosted on Hyper-V on a Windows Server over a TalkTalk business connection that is already successfully hosting a Postfix mailserver.
I would appreciate any ideas on this. I have tried steps such as disabling http2 connections back to http1.1 and also disabling a rewrite rule I added to direct http traffic to https but nothing I try makes any difference and the old server had been running fine for over a year.
This is a snippet from the error.log:
[Fri May 12 06:58:23.885767 2023] [proxy_fcgi:error] [pid 99667:tid 140182967776960] (70007)The timeout specified has expired: [client 198.204.231.122:50058] AH01075: Error dispatching request to : (polling)
[Fri May 12 07:01:03.251102 2023] [proxy_fcgi:error] [pid 99668:tid 140183328556736] (70007)The timeout specified has expired: [remote 10.10.1.1:52874] AH01075: Error dispatching request to : (polling)
[Fri May 12 07:01:08.525998 2023] [proxy_fcgi:error] [pid 99667:tid 140183126714048] (70007)The timeout specified has expired: [client 198.204.231.122:50156] AH01075: Error dispatching request to : (polling)
[Fri May 12 07:09:02.501388 2023] [proxy_fcgi:error] [pid 99668:tid 140182581909184] (70007)The timeout specified has expired: [client 198.204.231.122:50354] AH01075: Error dispatching request to : (polling)
[Fri May 12 07:11:45.345774 2023] [proxy_fcgi:error] [pid 99874:tid 140182288328384] (70007)The timeout specified has expired: [client 198.204.231.122:50451] AH01075: Error dispatching request to : (polling)
[Fri May 12 07:17:07.597795 2023] [proxy_fcgi:error] [pid 99668:tid 140182892308160] (70007)The timeout specified has expired: [client 198.204.231.122:50702] AH01075: Error dispatching request to : (polling)
[Fri May 12 07:17:09.431865 2023] [proxy_fcgi:error] [pid 99667:tid 140182883849920] (70007)The timeout specified has expired: [client 198.204.231.122:50801] AH01075: Error dispatching request to : (polling)
[Fri May 12 08:05:05.369726 2023] [proxy_fcgi:error] [pid 99667:tid 140182909028032] (70007)The timeout specified has expired: [client 52.167.144.160:47424] AH01075: Error dispatching request to : (polling)
[Fri May 12 08:05:09.001276 2023] [proxy_fcgi:error] [pid 99874:tid 140182288328384] (70007)The timeout specified has expired: [client 66.249.73.196:62107] AH01075: Error dispatching request to : (polling)
[Fri May 12 08:50:36.883724 2023] [proxy_fcgi:error] [pid 99667:tid 140182959384256] (70007)The timeout specified has expired: [client 66.249.76.190:49970] AH01075: Error dispatching request to : (polling)
[Fri May 12 09:11:55.907282 2023] [proxy_fcgi:error] [pid 99874:tid 140182254757568] (70007)The timeout specified has expired: [client 66.249.76.65:57755] AH01075: Error dispatching request to : (polling)
[Fri May 12 09:14:36.037940 2023] [proxy_fcgi:error] [pid 100060:tid 140182632265408] (70007)The timeout specified has expired: [client 194.38.21.214:60500] AH01075: Error dispatching request to : (polling)
This is a snippet from the access log:
194.38.21.214 - - [12/May/2023:08:26:35 +0000] "GET /admin/elfinder/php/connector.minimal.php HTTP/1.1" 302 609 "-" "ALittle Client"
157.55.39.224 - - [12/May/2023:07:38:37 +0000] "GET /courthouse/86/9826/1110397 HTTP/2.0" 200 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
157.55.39.224 - - [12/May/2023:07:38:37 +0000] "GET /courthouse/86/9826/1110397 HTTP/2.0" 500 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
52.167.144.162 - - [12/May/2023:07:43:49 +0000] "GET /frayproof=39571&bRVjuhw=853552 HTTP/2.0" 200 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
52.167.144.162 - - [12/May/2023:07:43:49 +0000] "GET /frayproof=39571&bRVjuhw=853552 HTTP/2.0" 500 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
114.119.139.205 - - [12/May/2023:08:39:55 +0000] "GET /ebay/QL320?C=N%3BO%3DD HTTP/1.1" 302 585 "https://www.rwsbetas.com/ebay/QL320?C=M%3BO%3DA" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
66.249.76.190 - - [12/May/2023:08:02:28 +0000] "GET /s/W2204874 HTTP/1.1" 504 4912 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
114.119.152.221 - - [12/May/2023:08:56:01 +0000] "GET /WiiKey/ HTTP/1.1" 302 555 "https://www.rwsbetas.com" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
66.249.76.65 - - [12/May/2023:08:23:51 +0000] "GET /s/A400738 HTTP/1.1" 504 4916 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.142 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
54.153.87.12 - - [12/May/2023:09:14:29 +0000] "GET /geeta HTTP/1.1" 301 4905 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
194.38.21.214 - - [12/May/2023:08:29:08 +0000] "GET /admin/elfinder/php/connector.minimal.php HTTP/1.1" 504 4929 "-" "ALittle Client"
10.10.1.1 - - [12/May/2023:09:25:50 +0000] "GET / HTTP/1.1" 302 541 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.0.0"
So it seems as if the request is just timing out when being handled by the PHP fast CGI module
Error dispatching request to : (polling)
The following steps were made in configuring apache2:
Installed packages:
apache2, php, php-curl, php-mbstring, php-imagick, php-intl
Enabled Apache2 modules:
suexec, authz_groupfile, headers, ssl, rewrite, proxy_http
Enabled PHP modules:
mbstring
PHP.ini changes:
upload_max_filesize = 32M
post_max_size = 32M
date.timezone = Europe/London
Lastly I switched over to HTTP2
sudo a2enmod http2
sudo apt install php-fpm
sudo a2dismod php8.1
sudo a2enconf php8.1-fpm
sudo a2enmod proxy_fcgi setenvif
sudo a2dismod php8.1
sudo a2dismod mpm_prefork
sudo a2enmod mpm_event
So looking
above
it would seem to be the proxy_fcgi module causing the issue. I will try reverting back to the older one.
Thanks
Robin