Score:0

docker health check for disk space not working as intended

in flag

I've got an nginx container which ends up with a full disk after it's been running for about 10 days. So if a new version of the app isn't released, errors start to occur that look like;

2022/01/15 22:45:04 [crit] 13#13: *406812 mkdir() "/var/cache/nginx/uwsgi_temp/9/07" failed (28: No space left on device) while reading upstream...

2022/01/15 22:44:37 [crit] 13#13: *406820 pwritev() "/var/cache/nginx/client_temp/0000001078" failed (28: No space left on device)...

This happened over the Christmas break so I thought the ideal situation here is to have the container health check ensure that there is free disk space. I thought I had achieved that with this container setup (but clearly not);

FROM nginx:1.21.5-alpine-perl

RUN apk update && \
    apk add --no-cache dnsmasq supervisor curl

COPY ./config/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
COPY ./config/nginx.conf /etc/nginx/nginx.conf

HEALTHCHECK --interval=15s --timeout=30s \
    CMD exit $(( $(df / | tail -n1 | awk '{print $5}' | sed 's/\%//') > 95 ? 1 : 0 )) || exit 1

How should I check for disk space in the health check?

Score:1
us flag

You should check your nginx configuration and make sure its cache size limits are configured properly according to the space you have available in the container.

in flag
The config doesn't actually specify any cache settings. From what I can see, `proxy_cache_path` is what I'd need to define. I've also read `client_body_buffer_size` (not currently defined) should match `client_max_body_size` which is currently 20mb
Score:1
in flag

Enter the container and check which directory inside it occupies the space using du (my first guess would be log files). Check if you can change the nginx configuration in a way that the container doesn't fill up the filesystem. If not, mount an external volume in the problematic directory.

in flag
That'd be ideal, but these containers are running on cloud formation stacks as fargate tasks. As far as I'm aware, it's not possible to connect to them.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.