For about 1-1.5 years ago I installed docker on a (vServer) Ubuntu 18.04 server and since then running many docker containers without any problems.
Yesterday I was applying a new update on one of my containers and as I tried to start the container I got the following error:
ERROR: for nextcloud Cannot start service nextcloud: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/memory/docker/e3efe26fdcbc055a6677b1c690bf837c771b728a9fb5ede2ab56c34c053c17b8: cannot allocate memory: unknown
I stopped all my containers and started to start one by one application (docker-compose):
after the 9th container started to get the error. Today I am not even able to start one single container!
I though that must be something wrong with the docker but soon found out that's not true.
If I try to create a folder in /sys/fs/cgroup/memory get the same error (can not allocate memory: unknown).
the system got still enough memory and disk space.
Filesystem Size Used Avail Use% Mounted on
/dev/ploop1234988 296G 65G 219G 23% /
none 4.0G 0 4.0G 0% /sys/fs/cgroup
none 4.0G 0 4.0G 0% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 1.2M 4.0G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
none 4.0G 0 4.0G 0% /run/shm
tmpfs 800M 0 800M 0% /run/user/1001
and free -m
total used free shared buff/cache available
Mem: 8000 103 7462 1 434 7895
Swap: 0 0 0
I've also checked some stats in the folder /sys/fs/cgroup/memory/
/sys/fs/cgroup/memory# cat memory.stat
rss_huge 0
mapped_file 8634368
shmem 49152
slab_unreclaimable 3182592
swap 0
cache 20856832
rss 1961984
slab_reclaimable 4808704
workingset_activate 0
pgpgin 21501
pgpgout 38697
pswpin 0
pswpout 0
pgfault 31728
pgmajfault 132
oom 0
inactive_anon 20480
active_anon 1990656
inactive_file 6459392
active_file 14348288
unevictable 0
hierarchical_memory_limit 8388608000
hierarchical_memsw_limit 8388608000
total_rss_huge 0
total_mapped_file 113991680
total_shmem 1208320
total_slab_unreclaimable 28409856
total_swap 0
total_cache 427483136
total_rss 69513216
total_slab_reclaimable 28618752
total_workingset_activate 73728
total_pgpgin 767407
total_pgpgout 1358271
total_pswpin 0
total_pswpout 0
total_pgfault 831603
total_pgmajfault 1647
total_oom 0
total_inactive_anon 143360
total_active_anon 70508544
total_inactive_file 191918080
total_active_file 234356736
total_unevictable 0
I've also checked some of the memory.kmem.* values but can not figure out what is going wrong and how to fix that problem by just checking the values?
Anybody here who already had this problem or knows the cause and solution of this problem?
Thank you in advance,
Gohlool
