Tomcat Version: 9.0.54
RHEL 7
We have several near identical application servers running Tomcat 9, and all but one are quick to startup. The last server, however, takes significantly longer for Tomcat to fully start.
It does not matter if we start using systemctl start tomcat or manually with calling catalina.sh start directly. Tomcat shows as started, and in the catalina.out log we see the Server initialization in x milliseconds message, and then nothing for about 15 minutes. Then it suddenly starts back like normal with Starting service [Catalina] and deploys the WARs just like all of the other servers. I updated the logging levels to FINEST, but there's still nothing additional in that wide gap, just adding a Setting state for [StandardServer[8005]] to [STARTING_PREP] before the hang, and then about 15 minutes later the Setting state for [StandardServer[8005]] to [STARTING]
On most of our servers, the time between those two log messages is about 2 seconds, on the one slow server it's around 12 minutes, 33 seconds on average
I have confirmed that we are using /dev/./urandom, verified free memory, swap space, disk space, and inodes. Cleared out the expanded WARs, cleared the work directory, restarted the server. Something is happening in between Server initialization and Starting service Catalina, but the logs are not telling me anything for that window of time. The fact that it is fairly reliably right around the same time limit, I'm thinking something is retrying until it reaches a timeout, but 12 and a half minutes seems like a bit of an odd timeout.
