We have a Tomcat application running on Elastic Beanstalk and our MySQL database is hosted on AWS RDS (2 or 3 t3.medium instances). Ever since we upgraded from MySQL 5 to MySQL 8 (currently 8.0.23), we've been having an issue that occurs about once a week. Most of the times the database is fine, but then, all of a sudden, the number of connections skyrockets (sometimes even surpassing the 307-connection limit during a 1-minute range, which is also something we don't get. How is it capable of going beyond that limit?) and that causes the Elastic Beanstalk instances to become degraded. Sometimes the entire database crashes after those connection peaks.
While monitoring the application's JVM with VisualVM, I've come to notice that, during those connection peaks, Tomcat suddenly creates dozens of worker threads. My guess is every one of those threads establishes a new connection to the database. While we could limit the number of those threads (the servers wouldn't be able to handle so many threads in the first place, after all), we want to understand what's causing that. Why is Tomcat creating so many threads and connections to our database? Is that a cause or a consequence of issues in the database? Where should we look to find the root of the issue?
I've Googled a lot, trying to find people who've had similar issues in order to shed a light on the problem. We've also tried analyzing the most expensive queries and other database performance insights but there seems to be no clear pattern.