What can you do when auditd halts the system?

I recently had an issue where my server powered off in the middle of running a script, seemingly randomly, but at about the same point each time, and then whenever I tried to power the server on again it would start the start up process and then when it got to a certain point just power itself off again before reaching the login options.

I initially thought it was something to do with the script and packages being installed, but taking a look at the security and CIS Benchmark documentation I found that the OS is installed with Level 2- Server configurations, to meet security requirements.

For what I need to do on my own server, I can apply a work around and edit the auditd.conf file and change this setting to be able to do what I need. However, in the production environment this workaround may not be appropriate or an allowed option.

I have a couple questions about this:

1. Can anything be done when the server reaches this state, as you can't even log on at this point, or is the only option to reimage server? (This is what I've been resorting to)

2. Presumably (I'm still trying to understand all the config options), this should not happen and the logs have some sort of rotate and retention policies, and I've just hit a corner case where what I need to do ends up filling these logs beyond what is the expected use case?

auditd is a Linux thing, of course.

Your organization should decide what action is appropriate should auditd decide it is critically out of space. Some environments value watching for security or integrity events so much, that a gap in logging is intolerable. Many others are not required to sacrifice availability for fewer gaps in audit logs.

As to your options, see man auditd.conf for config directives including admin_space_left_action. halt would explain your powering off host. single may be a useful compromise, stopping nearly all services, but allowing fixing the space problem on a console. Or merely report on the problem (syslog, email), lose data (rotate) or do nothing (ignore). Of course some of these mean you cannot claim to have done "Ensure system is disabled when audit logs are full".

Note that automated tools may configure the most conservative option, such as this Ansible playbook defaulting to halt.

Personally, I would be more suspect of an environment where halt is the policy but some hosts have outside the automation exceptions, compared to a policy that was honest and configured syslog. Although I am not a compliance person.

Always more than one way to solve things. Could boot a rescue distro or single user mode, then extend storage or purge logs to recover space.

All storage has the possibility to fill up. Find the root cause in this instance. Evaluate the necessity of rules, audit log space requirements has enormous variation depending on rules and workload. Improve your capacity planning and storage alerting to be reduce future space problems.