httpd quits after reload - slotmem-shm

My current httpd deployment setup involves generating new configuration files, relinking the config directory, then reloading the apache service. This process works fine for most of my hosts, however on of my proxy hosts, the httpd service will quit after issuing a reload.

Example:

/config/aaaa/*.conf
/config/bbbb/*.conf

/config/active -> /config/aaaa

And in my main httpd.conf file,

require /config/active/*.conf

The deployment process would remove the link, and relink a new config directory.

After issuing

systemctl reload httpd

The Error

I have the following error in my logs before the process just dies.

(28)No space left on device: AH02611: create: apr_shm_create(/etc/httpd/run/slotmem-shm-p975c3056_scanner_2.shm) failed

Debugging

I have plenty of disk space available:

[root@proxy3 log]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/cl_template-root  3.8G  3.4G  363M  91% /
tmpfs                         989M  102M  888M  11% /run
//10.3.36.10/config3          3.8G  2.6G  1.3G  68% /config

The directory, when the server is running, is only 6M.

[root@proxy3 ~]# du -h /etc/httpd/run/
0       /etc/httpd/run/htcacheclean
6.0M    /etc/httpd/run/

With ~1500 files

[root@proxy3 ~]# du -h /etc/httpd/run/* | wc -l
1521

Memories Available (should evict cache)

[root@proxy3 httpd]# free
              total        used        free      shared  buff/cache   available
Mem:        2025016      408128      113756      111216     1503132     1352584

My semaphores and ulimits seem fine

[root@proxy3 httpd]# sysctl -a | grep sem
kernel.sem = 32000      1024000000      500     32000

[root@proxy3 httpd]# ipcs -l

------ Messages Limits --------
max queues system wide = 32000
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18014398509481980
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 32000
max semaphores per array = 32000
max semaphores system wide = 1024000000
max ops per semop call = 500
semaphore max value = 32767

[root@proxy3 httpd]# ulimit
unlimited

Version Info

[root@proxy3 log]# httpd -v
Server version: Apache/2.4.37 (centos)
Server built:   May 20 2021 04:33:06

[root@proxy3 log]# uname -r
4.18.0-305.19.1.el8_4.x86_64

What else can I do to debug / correct this failure mode? My current workaround is to simply restart the service after it quits, however this is only a band-aid.

You have only 350Mio available on your / filesystem. Are you sure it's enought ? In all cases, consuming more than 90% of disk space should be a critical fact ;)

Examine the free space and size of the directory from the error message:

df -h /etc/httpd/run/
du -h /etc/httpd/run/*

Red Hat httpd package has a symlink /etc/httpd/run/ -> /run/httpd. Puts httpd DefaultRuntimeDir storage in a system tmpfs, without changing the convention of it living relative to ServerRoot. Note this means you should pay attention to /run.

What is log code 02611 anyway? Module mod_slotmem_shm which provides a shared memory API, Generic, so examine all modules loaded for if they use this to share state between workers. Approximate order of magnitude how large size can get, based on observations and capacity planning.

Regarding how to mitigate the problem, assuming a tmpfs is filling up, allocating a bit more memory to the host could give it more space. Or, you might tune your MPM to have fewer workers, or otherwise tweak the configuration to have less shared state. Finally, consider moving DefaultRuntimeDir to some temporary directory with sufficient space.

File backed shared memory maps are not System V shm you can see with ipcs. APR is portable and gives lots of options, but I think on this OS is file and mmap based.

On the topic of space, 363M available 91% used is not a lot free, even on a small host. Things like log files and package update transactions usually need more.