AWS RDS Slow Query, General, and Audit logs have disappeared. Enhanced Monitoring cannot be turned on

J. Gwinner

12/22/23, 1:18 AM

I had AWS RDS enhanced monitoring turned on in our Production and UAT RDS databases. These are connected to an ERP instance.

I went to look for slow queries while debugging a user issue, and the Audit, General, and Slow Query logs have disappeared.

They were and are enabled. They used to be in CloudWatch. Not a one left, even though we > 1 year retention.

If I go through the control panel and click on the hyperlink from "Published Logs" CloudWatch says

There was an error filtering log events
The specified log group does not exist

The error logs do exist, and do not show any logging errors.

I also tried turning the other logs off, letting settings persist, then turning them on. The other logs (slow SQL, etc) still don't exist.

These logs for production are business critical - I mean, if your audit log suddenly disappears, it's hard to build alerts, right? I can't believe there isn't even an error about these logs disappearing.

I did see that "Enhanced Monitoring" was turned off, so I turned it on, and got this:

Amazon RDS has been unable to configure enhanced monitoring on your instance: uat and this feature has been disabled.This is likely due to the rds-monitoring-role not being present and configured correctly in your account. Please refer to the troubleshooting section in the Amazon RDS documentation for further details.

The role does exist and hasn't changed since 2019.

rds-monitoring-role, Trust Relationships:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "monitoring.rds.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

One other thing: I did install DataDog's AWS integration CloudWatch routine about two weeks ago. That might be roughly when all the logs disappeared, but I think that's more coincidental, or I'd open a ticket with them. I deleted their CloudWatch stack, and the logs have not come back.

This question mentions log parameters, but I checked and ours haven't changed.

485

0 + 5

amazon-rds

amazon-cloudwatch

J. Gwinner

1/19/24, 11:44 PM

I've finally opened a support ticket, but AWS is also flummoxed. AWS Cloudwatch support is now talking to AWS RDS support.

J. Gwinner

1/30/24, 11:36 PM

AWS RDS support has basically gently informed me I have to turn on logging. Sigh. It's going to be a long service request.

Slim Fadi

5/4/24, 1:08 PM

any updates on this?

J. Gwinner

5/22/24, 4:25 PM

Unfortunately no - I'm no longer at that company and don't have access to that AWS ticket anymore! Long story.

J. Gwinner

7/18/24, 4:38 AM

AWS RDS support was generally useless, by the way. They kept telling me I had to turn on logging, even though it was, and I'd uploaded the screenshots showing the same. So I never got past the guy that just reads the book back at you (which often IS the problem,but sometimes it's not). I believe the error had something to do with an external monitoring package setup, via Firehose [sp?] but I couldn't prove it. I did think the product was good so I don't want to mention it by name.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: AWS RDS Slow Query, General, and Audit logs have disappeared. Enhanced Monitoring cannot be turned on

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.