Score:1

Server

Issues with AWS Aurora MySQL engine version "5.7.mysql_aurora.2.10.3"

Michael Engelbrecht

5/29/24, 7:58 AM

Is anyone else experiencing problems after recent automated AWS Aurora MySQL database engine updates? (specifically "5.7.mysql_aurora.2.10.3")

Over the weekend our web servers started reporting issues and were becoming non-responsive at times.
Troubleshooting today showed that since Thu morning we have been hitting max_connections=80 regularly on an AWS Aurora database that had been running very-lightly-loaded beforehand.

Further digging showed two step-changes in the DatabaseConnections metric (see plot 1)

The most recent step change on May 25 corresponds to an automated DB engine upgrade (5.7.mysql_aurora.2.10.3 --> 5.7.mysql_aurora.2.11.2) during the DB instance maintenance window...
...and an earlier step-change on May 7 corresponds to an unknown change during an earlier DB cluster maintenance window.

Other factors:

CPU usage has been impacted only slightly (see plot 2)
DB queries and usage patterns are unchanged.
There's nothing particularly interesting in the release-notes for this DB engine update.

We have put a few workarounds in place, but it looks like something in the recent Aurora MySQL DB engine update is adversely impacting performance, and I'm keen to hear from others who may be experiencing similar issues... and any solutions you may have found.

EDIT 6/6/2023

This Aurora DB seems to have now magically healed itself (ironically during another cluster maintenance window) - see plot below:

Note: based on analysis of our web servers, we had migrated a legacy web server from BlueHost (USA) to AWS (Sydney) to try to cope with the underperforming Aurora instance. This migration was implemented on June 2 and resulted in a step change improvement in performance... but then during the cluster maintenance window on June 4, the DB magically returned to pre-disruption performance once again!

Here's the final before/after profile:

I wish I could find a way to audit what is being done during each Aurora maintenance window!

522

2 + 9

mysql

amazon-rds

Rick James

5/29/24, 2:43 PM

Is there any kind of "connection pooling"? Do clients explicitly close there connection?

Wilson Hauck

5/29/24, 3:36 PM

Please post your code responsible for Connect, Process, Close activities for your clients success.

Wilson Hauck

5/29/24, 4:14 PM

@Michael-Engelbrecht What is result of SELECT @@innodb_buffer_pool_load_at_startup in your aurora2.11.2 version that uses more CPU?

Wilson Hauck

5/31/24, 11:49 AM

Please post your (PHP) code responsible for Connect, Process, Close activities that manages your clients connection.

Wilson Hauck

6/4/24, 1:17 AM

Additional DB information request from AWS AURORA, please. OS, Version? RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post TEXT data on justpaste.it and share the links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions.

Michael Engelbrecht

6/6/24, 6:05 AM

WilsonHauck - as mentioned earlier a LOT of different codebases are accessing this instance, so posting specific code is not practical... ...but I've just checked CloudWatch logs and it seems to have mysteriously healed itself during a cluster maintenance window - see edit above. Here's some answers to your questions: It's a single-instance Aurora MySQL db.t3.medium instance (2VCPU, 4GB RAM, etc) hosted in ap-southeast-2b zone. The results of the requested stats are here: https://justpaste.it/84njg

Score:0

Server

senrick

7/6/24, 7:26 AM

We are facing a similar issue as well, we upgraded our Aurora cluster from 2.11.1 to 2.11.2. We usually had around 30 - 50 sessions at a given time before the upgrade, but after the upgrade it's facing havoc. Session count has reached 300 +. If you are using 'INSERT IGNORE' and 'INSERT ON DUPLICATE KEY' statements, AWS has an internal fix that they will apply. There are performance issues definitely with the 2.11.2 version. We are still facing the issue, even upscaling the instance didn't resolve the issue.

+ 1

Quantim

7/11/24, 11:47 AM

This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://serverfault.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://serverfault.com/help/whats-reputation), you can also [add a bounty](https://serverfault.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/556094)

Score:0

Server

Daranporn Sabma

7/1/24, 4:19 PM

rest assured that you are not alone in this situation. Recently, I received an email from AWS urging me to upgrade my Aurora MySQL to engine 2.11.2 before the 30th of June.

However, after the upgrade, I have noticed a significant decline in the performance of various aspects, such as disk I/O and commit latency.

In an attempt to address this issue, I reached out to AWS support via email. Their response merely suggested to "ANALYZE TABLE." Regrettably, despite following their advice, the situation did not improve, and instead, it continues to cause numerous problems.

+ 1

Dave M

7/8/24, 11:16 AM

This does not provide an answer to the question. Once you have sufficient [reputation](https://serverfault.com/help/whats-reputation) you will be able to [comment on any post](https://serverfault.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/late-answers/555622)

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Issues with AWS Aurora MySQL engine version "5.7.mysql_aurora.2.10.3"

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.