Join Log in
Score:0
ujjwal avatar Ubuntu

Server and Apache getting shut down automatically/randomly

bo flag
ujjwal

I have a AWS ec2 server, I'm running 2 websites there on 2 different domains (obviously) using Apache2. One is domain.com, another is docs.domain.com.

I'm running a wordpress website on domain.com. I'm running react/next js app on docs.domain.com using pm2 & yarn.

Vhost for domain.com

<VirtualHost *:443>
    ServerName domain.com
    DocumentRoot /var/www/html
    ServerAlias domain.com
    ErrorLog /var/www/error.log
    CustomLog /var/www/requests.log combined

SSLEngine on
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/domain.com-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com-0001/privkey.pem
</VirtualHost>

<VirtualHost *:80>
    ServerName domain.com
    DocumentRoot /var/www/html
    ServerAlias bangdb.com
    ErrorLog /var/www/error.log
    CustomLog /var/www/requests.log combined
RewriteCond %{SERVER_NAME} =www.domain.com [OR]
RewriteCond %{SERVER_NAME} =domain.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

Vhost for docs.domain.com

<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName docs.domain.com

  ProxyRequests Off
  ProxyPreserveHost On
  ProxyVia Full
  ServerSignature Off

  <Proxy *>
      Require all granted
  </Proxy>

  ProxyPass / http://127.0.0.1:3000/
  ProxyPassReverse / http://127.0.0.1:3000/

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/domain.com-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com-0001/privkey.pem
</VirtualHost>
</IfModule>

Suddenly on 22 Jan 2023 around 07:39 (time in the logs) my server got shut down, both websites got down, and I was not able to login to my server via ssh also. I had to restart from AWS console to get it working again.

Here is the syslog when it got down

Jan 22 07:09:36 ip-172-30-0-37 systemd[1]: Starting Clean php session files...
Jan 22 07:09:36 ip-172-30-0-37 systemd[1]: Started Clean php session files.
Jan 22 07:17:00 ip-172-30-0-37 snapd[15896]: autorefresh.go:540: Cannot prepare auto-refresh change due to a permanent network error: persistent network error: Post https://api.snapcraft.io/v2/snaps/refresh: dial tcp: lookup api.snapcraft.io: Temporary failure in name resolution
Jan 22 07:17:00 ip-172-30-0-37 snapd[15896]: stateengine.go:149: state ensure error: persistent network error: Post https://api.snapcraft.io/v2/snaps/refresh: dial tcp: lookup api.snapcraft.io: Temporary failure in name resolution
Jan 22 07:17:01 ip-172-30-0-37 CRON[30597]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 22 07:20:01 ip-172-30-0-37 CRON[30603]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/lib/sm.bin/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jan 22 07:20:01 ip-172-30-0-37 sm-msp-queue[30622]: My unqualified host name (ip-172-30-0-37) unknown; sleeping for retry
Jan 22 07:21:01 ip-172-30-0-37 sm-msp-queue[30622]: unable to qualify my own domain name (ip-172-30-0-37) -- using short name
Jan 22 07:39:01 ip-172-30-0-37 CRON[30649]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jan 22 07:39:36 ip-172-30-0-37 systemd[1]: Starting Clean php session files...
Jan 22 07:39:36 ip-172-30-0-37 systemd[1]: Started Clean php session files.
Jan 22 07:39:47 ip-172-30-0-37 systemd[1]: Received SIGINT.
Jan 22 07:39:47 ip-172-30-0-37 systemd[1]: Stopped target Cloud-init target.
Jan 22 07:39:47 ip-172-30-0-37 systemd[1]: Stopping Authorization Manager...
Jan 22 07:39:47 ip-172-30-0-37 systemd[1]: Stopping User Manager for UID 1000...
Jan 22 07:39:47 ip-172-30-0-37 systemd[1]: Stopped target Timers.
Jan 22 07:39:47 ip-172-30-0-37 systemd[18626]: Stopped target Default.
Jan 22 07:40:41 ip-172-30-0-37 systemd[1]: Mounted POSIX Message Queue File System.
Jan 22 07:40:41 ip-172-30-0-37 systemd[1]: Started Set the console keyboard layout.
Jan 22 07:40:41 ip-172-30-0-37 systemd[1]: Mounted Huge Pages File System.
Jan 22 07:40:41 ip-172-30-0-37 systemd[1]: Started Create list of required static device nodes for the current kernel.
Jan 22 07:40:41 ip-172-30-0-37 systemd[1]: Mounted Kernel Debug File System.
Jan 22 07:40:41 ip-172-30-0-37 systemd[1]: Started Remount Root and Kernel File Systems.

Apache logs for the same time

[Sat Jan 21 06:25:02.412725 2023] [ssl:warn] [pid 31096] AH01909: ip-172-30-0-37.ec2.internal:443:0 server certificate does NOT include an ID which matches the server name
[Sat Jan 21 06:25:02.413374 2023] [mpm_prefork:notice] [pid 31096] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Sat Jan 21 06:25:02.413383 2023] [core:notice] [pid 31096] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jan 21 17:07:54.107574 2023] [mpm_prefork:error] [pid 31096] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Sun Jan 22 06:25:01.992892 2023] [mpm_prefork:notice] [pid 31096] AH00171: Graceful restart requested, doing restart
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using fe80::8b1:5dff:fe04:4731. Set the 'ServerName' directive globally to suppress this message
[Sun Jan 22 06:25:02.073408 2023] [ssl:warn] [pid 31096] AH01909: fe80::8b1:5dff:fe04:4731:443:0 server certificate does NOT include an ID which matches the server name
[Sun Jan 22 06:25:02.074116 2023] [mpm_prefork:notice] [pid 31096] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Sun Jan 22 06:25:02.074125 2023] [core:notice] [pid 31096] AH00094: Command line: '/usr/sbin/apache2'
[Sun Jan 22 07:39:47.609420 2023] [mpm_prefork:notice] [pid 31096] AH00169: caught SIGTERM, shutting down

I'm not able to figure out whats the issue, it has happened before on 17th Dec 2022, so it's the second time it has happened.

OS Details :- Ubuntu 18.04.6 LTS

422
0 + 3
in flag
matigo
The `server reached MaxRequestWorkers setting` item in your Apache error log is rather clear: the websites require more system resources than are currently permitted. You will want to update that `MaxRequestWorkers` setting to something *slightly* higher and test your system performance. Be careful not to go too high, otherwise you will run into contention issues as too many threads fight for the same limited resources, which will result in excessive system load and locking up.
1
Reply
ujjwal avatar
bo flag
ujjwal
Should I switch to Nginx as it is not a thread-based web server?
0
Reply
ujjwal avatar
bo flag
ujjwal
Can it be bot attack also, because the daily traffic is usually 50-60
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.