Join Log in
Score:2
MeSo2 avatar Server

Apache 2.4 on Windows slow to respond to initial first request

in flag
MeSo2

I started serving pre-compressed Brotli files on my website https://www.filmfix.com/en/home/. They work; but ever since then, Apache is having response issues along all VirtualHost setups (not just for the VirtualHost dedicated to serving these pre-compressed static HTML files.)

If I click around right after the initial (very slow) loading, it usually remains responsive (or if I just restarted Apache) -- but if I wait a bit, it stops being responsive. And may take from 7 seconds to 45 seconds to reconnect. On occasion it would even time out the browser, waiting to connect.

What could be causing this; and how do I fix it?

My https://www.webpagetest.org results look like this. webpagetest.org result

I do hope it is not related to my question from not too long ago: Internal Network drops connection.

I found a similar question on stackoverflow.com which gave me some more ideas, but nothing fixed it.

Or could it be of my own doing? I am processing batches of 18 asynchronous calls from three different domain names, generating HTML pages that get minimized and pre-compressed and pushed out to other servers. The CPU hovers around 80% at that. Could it be that I am overloading Apache? We are dealing with about 5,000 pages.

cpu usage

From my httpd-default.conf file

Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 40

# reqtimeout module is disabled
#<IfModule reqtimeout_module>
#  RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
#</IfModule>

These are the different browser symptoms I observe:

Chrome Timing Initial Connection/SSL

... and what I read when the connection is hanging:

(no information)

Chrome Timing

Microsoft Edge Waterfall

... and what I read when the connection is hanging:

Establishing secure connection ...

Microsoft Edge

FireFox Timings Blocked

... and what I read when the connection is hanging:

Performing TLS handshake with (my static. sub domain name)

or

Transferring data from connect.facebook.net

FireFox Timing Blocked

And I found that sometimes the Blocked time is just about as long as the TLS Setup time:

TLS Setup time

My Setup

Server: Apache/2.4.39 (Win64) OpenSSL/1.1.1c PHP/8.1.10, I do have 48GB of RAM available and the CPU is a i7-8700 @ 3.20GHz.

My DNS TTL is set to 3 hours, as I am about to re-locate files to different servers.

My httpd-ssl.conf file

SSLSessionCacheTimeout  300
SSLUseStapling Off
HostnameLookups Off
EnableSendfile Off 
EnableMMAP Off

<VirtualHost *:443>
    Protocols h2 h2c http/1.1
    ...
    SSLEngine on
    ...
    SSLHonorCipherOrder on
    SSLProtocol ALL -SSLv2 -SSLv3 -TLSv1 -TLSv1.1 
    SSLCipherSuite ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
</VirtualHost>

I tried setting <VirtualHost *:443> to <VirtualHost 0.0.0.0:443> (I am using only IPv4), but that did not help.

I turned off my Firewall to see if it is Firewall related, it is not.

running this

openssl s_client -connect www.filmfix.com:443 -status -servername www.filmfix.com

returns this

CONNECTED(000001B8)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.my_domainname_dot_com
verify return:1
OCSP response: no response sent
---
Certificate chain
 0 s:CN = *.my_domainname_dot_com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
  ...
-----END CERTIFICATE-----
subject=CN = *.my_domainname_dot_com

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5026 bytes and written 406 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 3D6884662...
    Session-ID-ctx:
    Resumption PSK: 5EA6E2B7D...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 ... 3i.

    Start Time: 1691621025
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 9BFE3...
    Session-ID-ctx:
    Resumption PSK: 38FFBF004D...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - a9 01 cb ...   a2 40   ....)..d...!.4.@

    Start Time: 1691621025
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

httpd -D DUMP_RUN_CFG

C:\64bit\Apache24\bin>httpd -D DUMP_RUN_CFG
ServerRoot: "C:/64bit/Apache24"
Main DocumentRoot: "C:/64bit/htdocs"
Main ErrorLog: "E:/log-files/apache/error_.log"
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="C:/64bit/Apache24/logs/" mechanism=default
PidFile: "C:/64bit/Apache24/logs/httpd.pid"
Define: DUMP_RUN_CFG

Any help would be very much appreciated.

Thank you.

251
1 + 2
ssl
Romeo Ninov avatar
in flag
Romeo Ninov
Do you have `HostnameLookups` set to `on` in apache config?
0
Reply
MeSo2 avatar
in flag
MeSo2
@RomeoNinov it is set to `HostnameLookups Off`
1
Reply
Score:1
MeSo2 avatar Server
in flag
MeSo2

I am keeping my fingers crossed.

I updated Apache -- going from 2.4.39 to 2.4.57.

In order to do that I had to first make sure Visual C++ Redistributable is updated:

You must first install the Visual C++ Redistributable for Visual Studio 2015-2022 x64.

Download and Install, if you have not done so already, see:

https://www.apachelounge.com/download/

Running the update I did a repair, followed by updating Apache.

The website looks and acts a bit different using the same Apache configuration files as before, and hopefully my initial connection speed issue resolved itself.

I also added default index files

DirectoryIndex static_html/com___en___home.min.html.br

And if all ends up to work well the update allows me to consider using HTTP/3. ChatGTP suggest:

As of my last update in September 2021, enabling HTTP/3 support in Apache on Windows requires a bit more manual effort compared to some other platforms due to the lack of precompiled packages. Here's a general guideline on how to install and enable the mod_http3 module for Apache on Windows:

  1. Download Required Software:

  2. Download and Build libnghttp2:

    • Download the latest stable release of libnghttp2 from https://github.com/nghttp2/nghttp2/releases.
    • Extract the downloaded archive to a directory.
    • Open the "Command Prompt for Visual Studio" from the Start menu.
    • Navigate to the directory where you extracted libnghttp2.
    • Run the following commands to build libnghttp2: 
      mkdir build
cd build
cmake -G "NMake Makefiles" ..
nmake
nmake install

  3. Download and Build Apache with HTTP/3 Support:

    • Download the Apache source code from https://httpd.apache.org/download.cgi.
    • Extract the downloaded archive to a directory.
    • Navigate to the srclib directory within the extracted Apache source directory.
    • Clone the quiche repository using Git: 
      git clone --recursive https://github.com/cloudflare/quiche
    • Rename the quiche directory to quic: 
      move quiche quic
    • Navigate to the root directory of the Apache source code.
    • Run the following commands to build Apache with HTTP/3 support: 
      configure --enable-ssl --enable-http2 --with-openssl=path\to\openssl
nmake

  4. Configure and Enable mod_http3:

    • Once Apache is built, navigate to the modules directory within the Apache source directory.
    • Copy the mod_http3 module file (mod_http3.so) to your Apache modules directory.
    • Open your Apache configuration file (httpd.conf) in a text editor.
    • Add the following lines to enable and configure mod_http3: 
      LoadModule http3_module modules/mod_http3.so

# Enable HTTP/3
<IfModule http3_module>
    AddHttp3Protocol h3-23
</IfModule>

  5. Start Apache with HTTP/3 Support:

    • Start Apache by running httpd.exe from the bin directory of the built Apache.
    • You might need to adjust your firewall settings to allow Apache to listen on the required ports.

Please note that this process involves building Apache and its dependencies from source, which can be complex and might require familiarity with software development tools. Since software and procedures can change over time, I recommend referring to the official Apache documentation and any relevant online resources for the most up-to-date instructions and guidance.

+ 4
Pimp Juice IT avatar
ch flag
Pimp Juice IT
Nice!! Be sure to make backup or do the upgrade test in a clone dev environment first if you can. You might find the ChatGPT steps need some tweaks or maybe does not work even though it sounds like it could. I'd do some more reading/searching of posts too: https://stackoverflow.com/questions/60324166/is-there-any-way-to-implement-http-3-quic-in-apache-http-server, https://www.reddit.com/r/apache/comments/o6a86x/why_is_apache_failing_to_implement_quic_http3/?onetap_auto=true, https://www.apachelounge.com/viewtopic.php?p=41699
1
Reply
Pimp Juice IT avatar
ch flag
Pimp Juice IT
I noticed you have Let's Encrypt SSL so wanted to mention might be worth a read on changes they make and/or are going to be making just in case that has any impact on this issue as well depending on your configuration, etc. for example: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/, https://letsencrypt.org/2023/07/10/cross-sign-expiration.html, and https://letsencrypt.org/certificates/... Just in case cert chains or any of that has impact on other firewalls, etc. along the hops on your side regardless of browser perhaps.
1
Reply
MeSo2 avatar
in flag
MeSo2
@PimpJuiceIT Good links to have! Regarding https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ it suggested *they must use version 1.1.0 or later* -- but my previews OpenSSL product version was 1.1.1c; and now was updated to 3.1.2. **The update made Apache much more responsive.**
1
Reply
MeSo2 avatar
in flag
MeSo2
something to note is that it takes Apache now a lot longer to restart `httpd -k restart`
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.