Join Log in
Score:0
dm90 avatar Server

How to avoid data loss during log rotation in linux/kubernetes?

in flag
dm90

Brief
We are running our workloads in kubernetes on AWS EKS. All our applications write the logs in /var/log/app_logs directory using hostPath. (yes it's not a best practice, this is due to legacy codebase, eventually we want to move to STDOUT).

Problems we face:

  1. We rely on the hourly logrotate to rotate the files under /var/log/app_logs
  2. During the hourly boundary, we tend to lose the loglines
  3. Upon investigating seems like the copytruncate is creating this datalos
  4. Looks like the application is not able to write to the logfile during the logrotation and we are losing those logs during the rotation.
  5. Since these services are running in kubernetes containers/pods, we won't be able to use the stop service, rotate and start the service approach.

The EKS worker nodes are running in Amazon Linux(centos).

The logrotate.conf we use:

"/var/log/app_logs/*.log" {
  create 664 foo foo
  daily
  missingok
  copytruncate
  rotate 7
  compress
  delaycompress
  notifempty
  sharedscripts
  postrotate
    /bin/bash /opt/scripts/upload-logs.sh
  endscript
  }

Is there anything that we could do to avoid this dataloss by tweaking the logrotate config? or is there any similar tools that solves this problem in kubernetes based environments.

189
0 + 0
user9517 avatar
cn flag
user9517
https://serverfault.com/questions/189477/rotate-logs-of-a-dumb-non-interactive-application/189880#189880
0
Reply
dm90 avatar
in flag
dm90
@user9517 Thanks for your response! But the problem in this (our) setup is, we don't have just couple of process in linux server. All these logs are from kubernetes pods. We may not be able to send the kill signal to the pod and in that case it would make it more complex by killing the pod and eventually it will get rescheduled somewhere else.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.