Score:1

OOM killer beeing invoked by snapd systemd timer, CPU usage locks at 100% (snapd, mount, certbot)

nc flag

Im having trouble debugging why my nodeJS webapp gets killed by the OOM killer in Debian 11. Im working on a small VPS (VMware ESX 7.0.3 build-21313628) by IONOS with single core 1GHz cpu, 512MB RAM, 1,9GB swap and 10GB SSD.

The webapp runs with pm2 and nginx as a reverse proxy, im using certbot systemd timer for automatic SSL renewals.

The issue which occurs is the following: The VPS gets unresponsive and i get a "critical ping" alert. The stats in the IONOS dashboard most of the time show cpu usage going to 100% and staying there, as well as a drop in RAM usage and no more responsiveness when pinging. It happens randomly, sometimes after a week or two of running, sometimes after 2 days. A software reboot of the server helps to resolve the issues. In normal operation, the RAM usage is at ~15-30% and CPU usage is at ~10%.

I took a look at the logs from the OOM killer and it seems that the issue always happens when mounting for either snapd itself or certbot.

localhost systemd[1]: Mounting Mount unit for certbot, revision 3024...
mount invoked oom-killer

See the full logs attached below:

/var/log/syslog.1-May  2 16:10:02 localhost snapd[749]: storehelpers.go:769: cannot refresh: snap has no updates available: "certbot", "core20"
/var/log/syslog.1-May  2 16:10:11 localhost systemd[1]: Reloading.
/var/log/syslog.1-May  2 16:10:12 localhost systemd[1]: Mounting Mount unit for snapd, revision 19122...
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.827965] mount invoked oom-killer: gfp_mask=0x40cc0(GFP_KERNEL|__GFP_COMP), order=3, oom_score_adj=0
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.828070] CPU: 0 PID: 34164 Comm: mount Not tainted 5.10.0-19-amd64 #1 Debian 5.10.149-2
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828102] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828144] Call Trace:
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828275]  dump_stack+0x6b/0x83
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828312]  dump_header+0x4a/0x1f4
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828335]  oom_kill_process.cold+0xb/0x10
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828379]  out_of_memory+0x1bd/0x4e0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828410]  __alloc_pages_slowpath.constprop.0+0xbcc/0xc90
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828439]  __alloc_pages_nodemask+0x2de/0x310
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828464]  kmalloc_order+0x28/0x100
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828486]  kmalloc_order_trace+0x19/0x90
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828520]  xz_dec_lzma2_create+0x21/0x80
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828543]  xz_dec_init+0x54/0xa0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828574]  squashfs_xz_init+0x3d/0x60 [squashfs]
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828597]  squashfs_decompressor_create+0x75/0x110 [squashfs]
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828623]  squashfs_decompressor_setup+0x82/0x180 [squashfs]
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828658]  squashfs_fill_super+0x338/0x8d0 [squashfs]
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828713]  ? init_once+0x10/0x10 [squashfs]
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828747]  get_tree_bdev+0x16d/0x260
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828771]  vfs_get_tree+0x22/0xc0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828793]  path_mount+0x458/0xa70
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828820]  __x64_sys_mount+0x107/0x140
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828848]  do_syscall_64+0x30/0x80
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828875]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828910] RIP: 0033:0x7f7fdc52e62a
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.828930] Code: 48 8b 0d 69 18 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 36 18 0d 00 f7 d8 64 89 01 48
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.829032] RSP: 002b:00007ffdb673a008 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.829139] RAX: ffffffffffffffda RBX: 00007f7fdc662264 RCX: 00007f7fdc52e62a
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.829184] RDX: 0000563cc5ff0c80 RSI: 0000563cc5ff0c60 RDI: 0000563cc5ff7f90
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.829223] RBP: 0000563cc5ff0a30 R08: 0000000000000000 R09: 00007f7fdc600be0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.829259] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000000
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.829287] R13: 0000563cc5ff7f90 R14: 0000563cc5ff0c80 R15: 0000563cc5ff0a30
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.829323] Mem-Info:
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.830356] active_anon:1169 inactive_anon:1417 isolated_anon:0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.830356]  active_file:734 inactive_file:980 isolated_file:0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.830356]  unevictable:0 dirty:12 writeback:6
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.830356]  slab_reclaimable:6041 slab_unreclaimable:47488
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.830356]  mapped:1105 shmem:21 pagetables:1039 bounce:0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.830356]  free:1799 free_pcp:0 free_cma:0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.834868] Node 0 active_anon:4676kB inactive_anon:5668kB active_file:2936kB inactive_file:3920kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4420kB dirty:48kB writeback:24kB shmem:84kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:5904kB all_unreclaimable? no
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.837038] Node 0 DMA free:1680kB min:96kB low:120kB high:144kB reserved_highatomic:0KB active_anon:16kB inactive_anon:44kB active_file:8kB inactive_file:4kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB pagetables:212kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.839128] lowmem_reserve[]: 0 388 388 388 388
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.839795] Node 0 DMA32 free:5516kB min:2472kB low:3088kB high:3704kB reserved_highatomic:2048KB active_anon:4660kB inactive_anon:5628kB active_file:2932kB inactive_file:3912kB unevictable:0kB writepending:32kB present:507776kB managed:434640kB mlocked:0kB pagetables:3944kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.841869] lowmem_reserve[]: 0 0 0 0 0
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.842599] Node 0 DMA: 9*4kB (U) 76*8kB (U) 65*16kB (UE) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1684kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.844121] Node 0 DMA32: 339*4kB (UMEH) 196*8kB (UMEH) 131*16kB (UMEH) 6*32kB (H) 1*64kB (H) 2*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5532kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.845672] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.846466] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.847312] 3047 total pagecache pages
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.848215] 1313 pages in swap cache
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.849004] Swap cache stats: add 16848324, delete 16847011, find 6908638/14622802
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.849850] Free swap  = 1846492kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.850673] Total swap = 1998844kB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.855457] Name                      Used          Total
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.856289] nf_conntrack             127KB        127KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.857122] ext4_system_zone           3KB          3KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.857977] scsi_sense_cache         296KB        296KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.858798] PINGv6                    30KB         30KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.859626] RAWv6                    339KB        339KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.860423] UDPv6                     31KB         31KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.861217] tw_sock_TCPv6             15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.862019] request_sock_TCPv6         15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.862763] TCPv6                     30KB         30KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.863490] mqueue_inode_cache         31KB         31KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.864215] UNIX                     200KB        200KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.864939] RAW                      512KB        512KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.865671] UDP                       31KB         31KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.866386] tw_sock_TCP               15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.867130] request_sock_TCP          15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.867852] TCP                      122KB        122KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.868540] hugetlbfs_inode_cache         31KB         31KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.869224] eventpoll_pwq             15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.869907] request_queue             63KB         63KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.870618] biovec-max               272KB        368KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.871278] biovec-128                32KB         32KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.871961] biovec-64                 32KB         32KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.872574] dmaengine-unmap-256         30KB         30KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.873165] dmaengine-unmap-128         31KB         31KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.873786] dmaengine-unmap-16        165KB        165KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.874333] skbuff_fclone_cache         32KB         32KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.874903] skbuff_head_cache        400KB        400KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.875429] file_lock_cache            7KB          7KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.875932] fsnotify_mark_connector          8KB          8KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.876455] net_namespace             30KB         30KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.876962] task_delay_info           51KB         51KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.877422] taskstats                 15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.877859] proc_dir_entry           149KB        149KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.878275] pde_opener                 3KB          3KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.878687] seq_file                   7KB          7KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.879065] shmem_inode_cache        759KB        759KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.879446] kernfs_node_cache       6076KB       6076KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.879814] mnt_cache                239KB        239KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.880184] filp                     322KB        448KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.880541] names_cache               64KB         64KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.880887] lsm_file_cache            51KB         51KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.881232] uts_namespace             15KB         15KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.881582] vm_area_struct           767KB        914KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.881925] mm_struct                 64KB         82KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.882267] files_cache              147KB        147KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.882621] signal_cache             441KB        441KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.882962] sighand_cache            682KB        744KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.883300] task_struct             2202KB       2315KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.883635] cred_jar                 102KB        102KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.883972] anon_vma_chain           205KB        268KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.884311] anon_vma                 226KB        304KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.884643] pid                       64KB         64KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.884975] Acpi-Operand             452KB        452KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.885308] Acpi-Parse                 7KB          7KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.885641] Acpi-State                 7KB          7KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.885968] numa_policy               47KB         47KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.886292] trace_event_file         158KB        158KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.886628] ftrace_event_field        398KB        398KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.886952] pool_workqueue            64KB         64KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.887277] task_group                74KB         74KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.887603] vmap_area                144KB        144KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.887931] kmalloc-8k               448KB        448KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.888256] kmalloc-4k            116840KB     116840KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.888579] kmalloc-2k              3808KB       3808KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.888900] kmalloc-1k              1733KB       1828KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.889218] kmalloc-512            14202KB      15024KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.889570] kmalloc-256             1056KB       1056KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.889888] kmalloc-192              252KB        252KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.890201] kmalloc-128              264KB        264KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.890515] kmalloc-96               200KB        200KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.890839] kmalloc-64               556KB        556KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.891149] kmalloc-32               480KB        480KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.891456] kmalloc-16               200KB        200KB
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.891767] kmalloc-8                 80KB         80KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.892063] kmem_cache_node           12KB         12KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.892357] kmem_cache                48KB         48KB
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.892681] Tasks state (memory values in pages):
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.892975] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.893310] [    337]     0   337    36649       67   274432      224          -250 systemd-journal
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.893639] [    352]     0   352     5461       93    69632      309         -1000 systemd-udevd
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.893969] [    419]   101   419    22110        0    73728      192             0 systemd-timesyn
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.894303] [    436]     0   436    12569        0    86016      368             0 VGAuthService
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.894661] [    437]     0   437    59708       33    90112      290             0 vmtoolsd
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.895006] [    640]     0   640    24972        5    73728      335             0 dhclient
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.895354] [    688]     0   688    24972        0    69632      375             0 dhclient
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.895696] [    744]     0   744     2319       23    53248       42             0 cron
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.896040] [    745]   104   745     1990       49    49152       95          -900 dbus-daemon
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.896392] [    748]     0   748    55199        0    81920      593             0 rsyslogd
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.896805] [    749]     0   749   184028      508   241664     4259          -900 snapd
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.897173] [    751]     0   751     3475       58    73728      213             0 systemd-logind
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.897541] [    753]     0   753   302239        0   274432     2909          -999 containerd
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.897934] [    757]     0   757     2094        0    49152       28             0 agetty
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.898339] [    761]     0   761     3339        8    65536      235         -1000 sshd
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.898731] [    785]     0   785   310713        0   364544     5526          -500 dockerd
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.899112] [   1024]  1000  1024     3859       23    69632      349             0 systemd
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.899496] [   1026]  1000  1026    25612        2   102400      763             0 (sd-pam)
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.899884] [   3877]  1000  3877   205117        1   942080     5054             0 PM2 v5.2.2: God
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.900278] [   4064]  1000  4064  2828374      423  1187840     7610             0 node /var/www/t
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.900679] [   5675]     0  5675    56818        0   176128     6802             0 nginx
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.901131] [  22752]    33 22752    56818        4   172032     6776             0 nginx
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.901546] [  34163]     0 34163     3167        5    65536       51          -900 systemctl
/var/log/syslog.1-May  2 16:10:13 localhost kernel: [168329.901963] [  34164]     0 34164     2392      139    61440       20             0 mount
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.902384] [  34166]     0 34166     5461      325    65536      250             0 systemd-udevd
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.902830] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=snap-snapd-19122.mount,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=node /var/www/t,pid=4064,uid=1000
/var/log/syslog.1:May  2 16:10:13 localhost kernel: [168329.903824] Out of memory: Killed process 4064 (node /var/www/t) total-vm:11313496kB, anon-rss:1692kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1160kB oom_score_adj:0

The systemd timer for certbot ssl renewal looks like this:

[Unit]
# Auto-generated, DO NOT EDIT
Description=Service for snap application certbot.renew
Requires=snap-certbot-3024.mount
Wants=network.target
After=snap-certbot-3024.mount network.target snapd.apparmor.service
X-Snappy=yes

[Service]
EnvironmentFile=-/etc/environment
ExecStart=/usr/bin/snap run --timer="00:00~24:00/2" certbot.renew
SyslogIdentifier=certbot.renew
Restart=no
WorkingDirectory=/var/snap/certbot/3024
TimeoutStopSec=30
Type=oneshot

Im not sure how to continue debugging at this point. I could not find any meaningful information online. Is the VPS just too small in terms of CPU and RAM or is there anything I can do about it to prevent the problem?

Andrew Henle avatar
ph flag
Disable memory overcommit and prevent [the out-of-fuel killer](https://lwn.net/Articles/104185/) from tossing passengers out of the airplane.... As a bonus you'll find the miscreant process that complains about not getting enough memory.
Andrew Henle avatar
ph flag
And according to my calculations, the 2828374 `total_vm` value for your `node` process means it's asked for 2828374 4kb pages of RAM, or 11,585,019,904 bytes. That's 11 GB on a system with 512 MB of RAM and under 2 GB of swap. That is not going to end well.
nr1chiefrocka avatar
nc flag
I will disable memory overcommit to see if that helps. I also saw that node has a high vm consumption, but in their github repo it says that this is practically not a problem, and that vm on x64 systems is theoretically unlimited: https://github.com/nodejs/node/issues/39499 If the system still runs out of memory because of snapd, and I can not remove it as a service, is there any other possible solution instead of increasing actual physical RAM available?
shodanshok avatar
ca flag
512 MB RAM is way too low for anything other than very basic services (eg: SSH jumphost). So yes - your VPS is simply too small.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.