How to limit log size on Elastic beanstalk with Corretto 8/Amazon Linux 2 platform?

We recently migrated one of our applications to the "Corretto 8 running on 64bit Amazon Linux 2/3.2.12" platform. Now we're seeing the issue that the log files are filling up our disk volume in a span of a bit over a week. We didn't have the issue before.

What's annoying is that we're streaming those logs to cloudwatch, so we really don't need them on the instance itself. What's even more annoying is that all the documentation, blog posts etc. I can find on the matter are apparently not written for this platform. When looking at the file structure, it seems like it's taking a different approach that I haven't completely figured out yet.

First of all, there's no /var/log/httpd ... folder or other folders for specific logs. So far I have figured out that what's sent to the web.stdout.log logstream (which on the old platform used to be named differently, but I don't remember how precisely. In any case, it's where the spring-boot application logs to) is stored in the file /var/log/messages.

I know this because while analysing why we were running out of disk space, this file would always be the single most largest on the entire disk. Except right after an instance started, then it had a reasonable size. After 3 days, it can reach a size of 2 GB, and it only keeps growing until there's nothing left. It is, without a doubt, the source of our trouble.

The question is, what can I do about it? Log rotation is in the default configuration, and I can see some files in /var/log/rotated (also a different folder from where rotated logs used to be), but those are tiny (the largest of them is an adorable 1.3 MB in size...), and the size of /var/log/messages never decreases, it only grows.

I have read numerous articles about how to solve similar problems by configuring logrotate using ebextensions, by AWS themselves and bloggers, but in all of them I can see very quickly that they are built on top of a very different structure, so they don't seem applicable. Also, we have experienced already during the migration that some things that were done with ebextensions were now not done with ebextensions anymore, but the documentation is rather lacking.

Does anybody know the platform enough to give me a clue about how I might go about configuring this? I don't even need to rotate the logs, strictly speaking, I just want them gone as soon as they're older than an hour or so. We have them on cloudwatch after all...

Additional information The deeper I'm looking into this, the more I'm getting confused. There is a /var/log/web.stdout.log file, which receives the same messages as /var/log/messages. Looking at the contents of logrotate.elasticbeanstalk.hourly, this is the file that's actually under log rotation, and that appears to be working fine.

Doing an lsof, the process using that file is rsyslogd. According to shearn89s comment below, apparently that's the new syslog and everything that logs to stdout gets logged there.

Realising that that file isn't under any kind of rotation, I'm trying to set that up. Just on the instance directly for now, I'll figure out later how to do it in the actual environment config.

I've created a new config file in /etc/logrotate/elasticbeanstalk.hourly with the following content:

/var/log/messages {
 su root root
 size 10M
 rotate 5
 missingok
 compress
 notifempty
 copytruncate
 dateext
 dateformat %s
 olddir /var/log/rotated
}

Running sudo /usr/sbin/logrotate /etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.messages.conf --debug, I get the following output:

rotating pattern: /var/log/messages  10485760 bytes (5 rotations)
olddir is /var/log/rotated, empty log files are not rotated, old logs are removed
considering log /var/log/messages
  log needs rotating
rotating log /var/log/messages, log->rotateCount is 5
Converted ' %s' -> '%s'
dateext suffix '1646905905'
glob pattern '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
copying /var/log/messages to /var/log/rotated/messages1646905905
truncating /var/log/messages
compressing log with: /bin/gzip

That looks promising, but the cold fact is that nothing seems to have changed. The file still has the same size, and no file was created in the rotation folder. It seems to have no effect at all. I've changed the configuration so it moves and creates instead of truncates in place, I've tried -f, all with the same result. There's no error from logrotate, but no effect either. That file seems bloody impervious.

I saved this in .ebextensions/liblogrotate.config:

files:
   /etc/remove_old_logs.sh:
     owner: root
     group: root
     mode: "000644"
     content: |
       #!/bin/sh
       find /var/log -type f -mtime +7 -exec rm {} +

container_commands:
  01_crontab:
    command: "( crontab -l | grep -v -F \". /etc/remove_old_logs.sh\" ; echo \"0 0 * * * . /etc/remove_old_logs.sh\" ) | crontab -"

It creates a crontab that removes files older than 7 days in the /var/log folder. I know it's not optimal, but I don't care that much about those logs.

The easiest way is to stop sending web stdout to /var/log/messages. It requires only the line stop to be added to the corresponding rsyslog config in /etc/rsyslog.d/web.conf:

# This rsyslog file redirects Elastic Beanstalk platform logs.
# Logs are initially sent to syslog, but we also want to divide
# stdout and stderr into separate log files.

if $programname  == 'web' then {
  *.=warning;*.=err;*.=crit;*.=alert;*.=emerg /var/log/web.stderr.log
  *.=info;*.=notice /var/log/web.stdout.log
  # THE LINE BELOW STOPS SENDING OUTPUT TO ANY OTHER LOG FILES
  stop
}

Create an executable file (with chmod +x) in the .platform/hooks/predeploy directory with the following contents to apply this change in configuration on deploy of a new version:

#!/bin/sh

FILE="/etc/rsyslog.d/web.conf"

CONTENTS=$(cat <<-END
# This rsyslog file redirects Elastic Beanstalk platform logs.
# Logs are initially sent to syslog, but we also want to divide
# stdout and stderr into separate log files.

if $programname  == 'web' then {
  *.=warning;*.=err;*.=crit;*.=alert;*.=emerg /var/log/web.stderr.log
  *.=info;*.=notice /var/log/web.stdout.log
  stop
}
END
)

echo "$CONTENTS" > "$FILE"
service rsyslog restart

If you look at /etc/logrotate.d/syslog, you will find that there is already a provision to rotate the file /var/log/messages. But the configuration does not compress the rotated files and it keeps up to 4 weeks of rotated logs. This is okay for light use but on a server app which stream logs for each access, this can quickly exhaust the disk.

AWS now recommends using .platform/hooks files instead of .ebextensions. also, .ebextensions will create a copy of the file with .bak extension if over-written. This might result in duplicate logrotate files which can result in error.

I have added the following configuration file in the path .platform/hooks/predeploy/modify-logrotate.sh

Make sure that this file is executable and starting with a proper shebang.

#!/bin/sh

fileName=/etc/logrotate.d/syslog
lineCount=($(wc -l "$fileName"))
lineToInsert="$((${lineCount[0]}))"
echo $lineToInsert
sed -i "${lineToInsert}i    compress\n    rotate 1\n    daily" "$fileName"

The original config file ends with } as the last line. The above code inserts extra configuration lines above the ending line so that the final code is valid.

I am not sure if this change can cause invalid syntax in some cases but I found no better way to modify this file in a safer way.