I have Apache/HTTPD server at GCP (VM Instance) with version
Server version: Apache/2.4.6 (CentOS)
Server built: Nov 16 2020 16:18:20
Since I was a developer, I don't advance with server configuration. So I have a problem, Our production server had 2 times Apache/HTTPD shutdown unexpectedly. I'm not sure what happened. After reading several article on internet (also digging information on SO and Google), I collect all possible logs issue and conclude maybe our server got DDOS attack. But still i'm not sure. I'm really need your help guys to figuring out what happened in our server.
The httpd service shutdown on 03:16 AM and I trying to up again at 07:10 AM. Here some logs. (if you need more logs, don't hesitate to let me know)
ssl_request_log
[12/Oct/2021:03:18:11 +0700] 114.122.15.78 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "POST /public/api/mobile/.../... HTTP/1.1" 1097
[12/Oct/2021:03:18:11 +0700] 114.122.15.78 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "POST /public/api/mobile/.../... HTTP/1.1" 277
[12/Oct/2021:03:18:11 +0700] 114.122.15.78 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "POST /public/api/mobile/.../... HTTP/1.1" 277
[12/Oct/2021:03:18:27 +0700] 114.122.15.78 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "POST /public/api/mobile/.../... HTTP/1.1" 467
[12/Oct/2021:07:48:04 +0700] 182.1.65.179 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 2513
[12/Oct/2021:07:48:07 +0700] 182.1.65.179 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /favicon.ico HTTP/1.1" -
[12/Oct/2021:08:04:47 +0700] 182.1.65.179 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET / HTTP/1.1" 2513
[12/Oct/2021:08:04:48 +0700] 182.1.65.179 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 "GET /favicon.ico HTTP/1.1" -
ssl_access_log
114.122.15.78 - - [12/Oct/2021:03:18:10 +0700] "GET /public/api/mobile/.../... HTTP/1.1" 200 112
114.122.15.78 - - [12/Oct/2021:03:18:11 +0700] "POST /public/api/mobile/.../... HTTP/1.1" 200 1097
114.122.15.78 - - [12/Oct/2021:03:18:11 +0700] "POST /public/api/mobile/.../... HTTP/1.1" 200 277
114.122.15.78 - - [12/Oct/2021:03:18:11 +0700] "POST /public/api/mobile/.../... HTTP/1.1" 200 277
114.122.15.78 - - [12/Oct/2021:03:18:27 +0700] "POST /public/api/mobile/..../... HTTP/1.1" 200 467
182.1.65.179 - - [12/Oct/2021:07:48:04 +0700] "GET / HTTP/1.1" 200 2513
182.1.65.179 - - [12/Oct/2021:07:48:07 +0700] "GET /favicon.ico HTTP/1.1" 200 -
182.1.65.179 - - [12/Oct/2021:08:04:47 +0700] "GET / HTTP/1.1" 200 2513
182.1.65.179 - - [12/Oct/2021:08:04:48 +0700] "GET /favicon.ico HTTP/1.1" 200 -
messages
Oct 12 03:20:01 xxx-server-1 systemd: Started Session 211514 of user apache.
Oct 12 03:21:01 xxx-server-1 systemd: Started Session 211515 of user apache.
Oct 12 03:22:01 xxx-server-1 systemd: Started Session 211516 of user apache.
Oct 12 03:23:01 xxx-server-1 systemd: Started Session 211517 of user apache.
Oct 12 07:09:31 xxx-server-1 kernel: Initializing cgroup subsys cpuset
Oct 12 07:09:31 xxx-server-1 kernel: Initializing cgroup subsys cpu
Oct 12 07:09:31 xxx-server-1 kernel: Initializing cgroup subsys cpuacct
Oct 12 07:09:31 xxx-server-1 kernel: Linux version 3.10.0-1127.19.1.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Tue Aug 25 17:23:54 UTC 2020
access_log
::1 - - [12/Oct/2021:02:12:16 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
186.33.94.58 - - [12/Oct/2021:02:15:16 +0700] "GET / HTTP/1.1" 301 234 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
78.128.112.14 - - [12/Oct/2021:02:17:15 +0700] "\x03" 400 226 "-" "-"
::1 - - [12/Oct/2021:02:21:11 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:22:27 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:30 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:31 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:32 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:33 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:37 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:38 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:25:41 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:26:01 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:30:51 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:30:52 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:30:53 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:32:50 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:43:28 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:43:29 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:43:30 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:43:54 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:52:35 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:52:36 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:53:11 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:54:25 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:55:43 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:55:44 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:56:09 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:02:59:49 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:03:00:33 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:03:00:34 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:03:00:42 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:03:04:46 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
::1 - - [12/Oct/2021:03:16:20 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.21 (internal dummy connection)"
182.1.93.51 - - [12/Oct/2021:07:10:49 +0700] "GET / HTTP/1.1" 301 234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
31.150.61.16 - - [12/Oct/2021:07:10:52 +0700] "GET / HTTP/1.1" 301 234 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
and what me think i got DDOS attack is on my access_log that show some foreign URL (correct me if i'm wrong). here is some sample
112.78.156.175 - - [12/Oct/2021:08:10:50 +0700] "GET /id/sains/teori-gravitasi-bertentangan-dengan-hukum-kekekalan-energi HTTP/1.1" 301 301 "https://www.bing.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Edg/94.0.992.38"
::1 - - [12/Oct/2021:08:10:59 +0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.24 (internal dummy connection)"
182.1.65.179 - - [12/Oct/2021:08:11:05 +0700] "GET / HTTP/1.1" 301 234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
132.145.247.182 - - [12/Oct/2021:08:11:20 +0700] "GET /id/ HTTP/1.0" 301 237 "http://www.rustamaji.net/id/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
error_log (i got nothing at eror_log)
[Mon Oct 11 23:47:59.036428 2021] [core:error] [pid 23856] [client 45.146.164.110:48008] AH00126: Invalid URI in request POST /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh HTTP/1.1
[Tue Oct 12 07:09:41.719194 2021] [suexec:notice] [pid 865] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
My question is what makes my Apache Server shutdown unexpectedly? And how to prevent it on GCP environment? I heard that cloud armor could help for DDOS attack (but since i'm not sure what cause the problem)
Please help. Thanks in advance. I'm really appreciate your help.
