Hi have a Tomcat 9 running in a AWS machine using the following docker compose file:
version: '3'
services:
fstomcat:
image: tomcat:9
container_name: fstomcat
ports:
- 443:443
volumes:
- /opt/tomcat/webapps:/usr/local/tomcat/webapps
- /opt/tomcat/conf:/usr/local/tomcat/conf
- /opt/tomcat/logs:/usr/local/tomcat/logs
There are no webapps at the moment (webapps is empty). This EC2 only has Tomcat. No Apache, no other web server or database server.
However, AWS reports random spikes in CPU usage. When I get into the container java is at 199% CPU. The latest spike happened on 2021-06-20 13:30 and the only logs I have for that day are:
catalina:
20-Jun-2021 09:45:04.595 INFO [https-openssl-nio-443-exec-6] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request target [/index.php?s=/Index/\think\app/invokefunction&function=call_user_func_$
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:490)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:261)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
access:
192.241.220.30 - - [20/Jun/2021:00:22:53 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:31:59 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:32:03 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:32:03 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:32:03 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:32:04 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:32:06 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:32:08 +0000] "-" 400 -
74.82.47.4 - - [20/Jun/2021:00:33:03 +0000] "-" 400 -
162.216.17.178 - - [20/Jun/2021:00:41:24 +0000] "-" 400 -
128.1.248.42 - - [20/Jun/2021:01:17:50 +0000] "GET / HTTP/1.1" 404 682
192.241.215.206 - - [20/Jun/2021:01:56:40 +0000] "GET /actuator/health HTTP/1.1" 404 682
45.33.79.16 - - [20/Jun/2021:02:19:19 +0000] "-" 400 -
209.17.97.98 - - [20/Jun/2021:02:57:39 +0000] "-" 400 -
162.216.17.71 - - [20/Jun/2021:04:19:13 +0000] "-" 400 -
45.83.67.150 - - [20/Jun/2021:04:58:00 +0000] "-" 400 -
66.240.205.34 - - [20/Jun/2021:06:08:25 +0000] "-" 400 -
45.33.79.16 - - [20/Jun/2021:06:18:56 +0000] "-" 400 -
162.62.123.46 - - [20/Jun/2021:08:04:09 +0000] "GET / HTTP/1.0" 404 682
192.241.218.53 - - [20/Jun/2021:08:12:25 +0000] "GET /owa/auth/logon.aspx?url=https%3a%2f%2f1%2fecp%2f HTTP/1.1" 404 682
162.216.17.71 - - [20/Jun/2021:08:18:54 +0000] "-" 400 -
45.146.165.123 - - [20/Jun/2021:09:44:57 +0000] "POST /Autodiscover/Autodiscover.xml HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:44:59 +0000] "GET /_ignition/execute-solution HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:00 +0000] "GET / HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:02 +0000] "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:04 +0000] "GET null HTTP/1.1" 400 2273
45.146.165.123 - - [20/Jun/2021:09:45:06 +0000] "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:06 +0000] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:07 +0000] "GET /wp-content/plugins/wp-file-manager/readme.txt HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:08 +0000] "POST /mifs/.;/services/LogService HTTP/1.1" 404 682
45.146.165.123 - - [20/Jun/2021:09:45:09 +0000] "GET /console/ HTTP/1.1" 404 682
45.33.79.16 - - [20/Jun/2021:10:19:25 +0000] "-" 400 -
193.118.53.210 - - [20/Jun/2021:10:20:10 +0000] "GET / HTTP/1.1" 404 682
162.216.17.71 - - [20/Jun/2021:12:19:00 +0000] "-" 400 -
138.68.175.207 - - [20/Jun/2021:13:28:31 +0000] "-" 400 -
138.68.175.207 - - [20/Jun/2021:13:28:35 +0000] "-" 400 -
138.68.175.207 - - [20/Jun/2021:13:28:35 +0000] "-" 400 -
138.68.175.207 - - [20/Jun/2021:13:28:35 +0000] "-" 400 -
138.68.175.207 - - [20/Jun/2021:13:28:35 +0000] "-" 400 -
165.22.86.42 - - [20/Jun/2021:13:56:50 +0000] "-" 400 -
162.142.125.39 - - [20/Jun/2021:14:01:08 +0000] "-" 400 -
162.142.125.39 - - [20/Jun/2021:14:01:10 +0000] "GET / HTTP/1.1" 404 682
162.142.125.39 - - [20/Jun/2021:14:01:10 +0000] "GET / HTTP/1.1" 404 682
60.217.75.69 - - [20/Jun/2021:14:22:42 +0000] "GET / HTTP/1.1" 404 682
172.105.172.151 - - [20/Jun/2021:14:35:22 +0000] "GET /owa/ HTTP/1.1" 404 682
192.241.214.26 - - [20/Jun/2021:15:04:40 +0000] "GET / HTTP/1.1" 404 682
34.90.100.245 - - [20/Jun/2021:15:18:59 +0000] "GET /.env HTTP/1.1" 404 682
34.90.100.245 - - [20/Jun/2021:15:19:00 +0000] "POST / HTTP/1.1" 404 682
128.14.134.170 - - [20/Jun/2021:16:01:33 +0000] "GET / HTTP/1.1" 404 682
97.107.132.27 - - [20/Jun/2021:16:19:28 +0000] "-" 400 -
173.255.234.116 - - [20/Jun/2021:16:30:04 +0000] "-" 400 -
23.90.160.130 - - [20/Jun/2021:16:37:09 +0000] "GET / HTTP/1.1" 404 682
23.95.191.195 - - [20/Jun/2021:16:50:06 +0000] "POST /GponForm/diag_Form?style/ HTTP/1.1" 404 682
162.216.17.71 - - [20/Jun/2021:18:18:33 +0000] "-" 400 -
193.118.53.210 - - [20/Jun/2021:18:29:39 +0000] "GET / HTTP/1.1" 404 682
51.159.23.43 - - [20/Jun/2021:18:44:34 +0000] "GET / HTTP/1.1" 404 682
45.79.168.6 - - [20/Jun/2021:20:19:38 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:48:59 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:48:59 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:48:59 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:48:59 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:48:59 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:48:59 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:49:00 +0000] "-" 400 -
192.241.220.87 - - [20/Jun/2021:20:49:00 +0000] "-" 400 -
192.241.212.36 - - [20/Jun/2021:21:03:09 +0000] "-" 400 -
128.14.209.162 - - [20/Jun/2021:21:36:20 +0000] "GET / HTTP/1.1" 404 682
192.241.218.97 - - [20/Jun/2021:22:11:38 +0000] "GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.1" 404 682
45.79.144.15 - - [20/Jun/2021:22:19:16 +0000] "-" 400 -
162.142.125.40 - - [20/Jun/2021:23:08:05 +0000] "-" 400 -
162.142.125.40 - - [20/Jun/2021:23:08:07 +0000] "GET / HTTP/1.1" 404 682
162.142.125.40 - - [20/Jun/2021:23:08:07 +0000] "GET / HTTP/1.1" 404 682
45.63.12.50 - - [20/Jun/2021:23:49:07 +0000] "-" 400 -
syslog:
Jun 20 13:00:24 ip-172-30-1-110 systemd-timesyncd[21286]: Network configuration changed, trying to establish connection.
Jun 20 13:00:24 ip-172-30-1-110 systemd-networkd[13629]: ens5: Configured
Jun 20 13:00:24 ip-172-30-1-110 systemd-timesyncd[21286]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Jun 20 13:17:01 ip-172-30-1-110 CRON[21362]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jun 20 13:30:24 ip-172-30-1-110 systemd-networkd[13629]: ens5: Configured
Jun 20 13:30:24 ip-172-30-1-110 systemd-timesyncd[21286]: Network configuration changed, trying to establish connection.
Jun 20 13:30:24 ip-172-30-1-110 systemd-timesyncd[21286]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
This is how the server starts:
23-Jun-2021 17:37:03.904 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [maxSpareThreads] to [75]
23-Jun-2021 17:37:03.999 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.41
23-Jun-2021 17:37:04.000 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Dec 3 2020 11:43:00 UTC
23-Jun-2021 17:37:04.001 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.41.0
23-Jun-2021 17:37:04.003 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
23-Jun-2021 17:37:04.003 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.0-1029-aws
23-Jun-2021 17:37:04.004 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
23-Jun-2021 17:37:04.004 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/local/openjdk-11
23-Jun-2021 17:37:04.005 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.10+9
23-Jun-2021 17:37:04.005 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
23-Jun-2021 17:37:04.006 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/local/tomcat
23-Jun-2021 17:37:04.006 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/local/tomcat
23-Jun-2021 17:37:04.007 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
23-Jun-2021 17:37:04.008 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
23-Jun-2021 17:37:04.008 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
23-Jun-2021 17:37:04.008 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
23-Jun-2021 17:37:04.009 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
23-Jun-2021 17:37:04.009 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
23-Jun-2021 17:37:04.010 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
23-Jun-2021 17:37:04.016 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
23-Jun-2021 17:37:04.017 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
23-Jun-2021 17:37:04.018 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
23-Jun-2021 17:37:04.018 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
23-Jun-2021 17:37:04.018 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
23-Jun-2021 17:37:04.025 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.5].
23-Jun-2021 17:37:04.025 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
23-Jun-2021 17:37:04.026 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
23-Jun-2021 17:37:04.030 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1d 10 Sep 2019]
23-Jun-2021 17:37:04.634 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-443"]
23-Jun-2021 17:37:05.225 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1569] milliseconds
23-Jun-2021 17:37:05.324 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
23-Jun-2021 17:37:05.324 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.41]
23-Jun-2021 17:37:05.365 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-openssl-nio-443"]
23-Jun-2021 17:37:05.396 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [170] milliseconds
We deactivated any automatic update in the EC2 machine to remove the possibility that an update in docker was causing this. But the only thing that we can do is restart it.
I would like to know if anyone has dealt with something like this and has an idea on how to correct it
