Kubernetes: How/where to log when Secrets are deleted?

Michael Martinez

7/6/24, 12:26 AM

Does kubernetes capture - in the event log or elsewhere - when Secrets are deleted? I'm not seeing this showing up in the event log: kubectl get events --field-selector involvedObject.kind=Secret --watch and I'm not finding it in /var/log/pods/kube_system-kube-apiserver/either.

We're using something called external-secrets-operator to manage secrets, and we would like to be informed when a secret is deleted, as this isn't supposed to happen with ESO, so it's an edge case that we'd like to monitor for.

110

1 + 3

linux

kubernetes

larsks

7/6/24, 3:41 AM

I presume -- but don't know for sure, hence the comment rather than answer -- that this would show up in the [audit logs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/).

Sai Chandra Gadde

7/7/24, 4:28 PM

Did you have time to check my answer? It helped you to solve your issue? If not, I am happy to assist further.<br>

Michael Martinez

7/8/24, 5:09 PM

yeah I checked. We don't have auditing enabled. Would have to enable it if we want these events.

Score:1

Server

Sai Chandra Gadde

7/6/24, 6:22 AM

Using event log kubectl get events --field-selector involvedObject.kind=Secret --watch you will get existing secrets not the deleted resources, for deleted resources you need to use Audit logs, audit logs are automatically enabled for some managed kubernetes like GKE and AWS. Managed Kubernetes services like EKS, AKS or GCP provide support by routing cluster audit logs into centralized logging services (like AWS CloudWatch,GCP logs explorer) easily.

To find out who deleted the namespace and at what time, it only gives the IAM user, below query is used to get when and who deleted the secret.

logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.secrets"
NOT protoPayload.methodName:"get"
NOT protoPayload.methodName:"list"
NOT protoPayload.methodName:"watch"

If you are not using any managed kubernetes, follow this blog written by Daniel Olaogun which elaborated on how to use the audit logs.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Kubernetes: How/where to log when Secrets are deleted?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.