I have a question about the OOM killer logs. We are expecting a lot of OOM kills.
The ecosystem
My ecosystem looks like below:
I have a server with 4 cores and 8 GB of RAM.
I am running there the following most resource-consuming programs:
- PostgreSQL database
- Two applications with processes called
vega - native binaries compiled from Go code.
I cannot isolate applications from PostgreSQL because the application uses PostgreSQL as internal storage and must share the file system.
OOM logs
I have the following logs in the syslog:
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: vega invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: CPU: 0 PID: 742994 Comm: vega Not tainted 5.4.0-122-generic #138-Ubuntu
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Hardware name: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Call Trace:
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: dump_stack+0x6d/0x8b
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: dump_header+0x4f/0x1eb
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: oom_kill_process.cold+0xb/0x10
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: out_of_memory+0x1cf/0x4d0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: __alloc_pages_slowpath+0xd5e/0xe50
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: __alloc_pages_nodemask+0x2d0/0x320
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: alloc_pages_current+0x87/0xe0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: __page_cache_alloc+0x72/0x90
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: pagecache_get_page+0xbf/0x300
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: filemap_fault+0x6b2/0xa50
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ? unlock_page_memcg+0x12/0x20
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ? page_add_file_rmap+0xff/0x1a0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ? xas_load+0xd/0x80
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ? xas_find+0x17f/0x1c0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ? filemap_map_pages+0x24c/0x380
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ext4_filemap_fault+0x32/0x50
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: __do_fault+0x3c/0x170
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: do_fault+0x24b/0x640
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: __handle_mm_fault+0x4c5/0x7a0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: handle_mm_fault+0xca/0x200
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: do_user_addr_fault+0x1f9/0x450
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: __do_page_fault+0x58/0x90
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: ? exit_to_usermode_loop+0xbf/0x160
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: do_page_fault+0x2c/0xe0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: do_async_page_fault+0x39/0x70
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: async_page_fault+0x34/0x40
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RIP: 0033:0x458f17
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Code: Bad RIP value.
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RSP: 002b:000000c03e2c5340 EFLAGS: 00010212
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RAX: 00000000016e4eb6 RBX: 0000000000000000 RCX: 000000000552ee80
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RDX: 000000000000f32b RSI: 0000000000019db8 RDI: 0000000004b89160
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: RBP: 000000c03e2c5350 R08: 000000000000f32c R09: 0000000004c02ac0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: R10: 0000000001c0a9e5 R11: 00000000051b6ab8 R12: 0000000000000000
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: R13: 0000000000000000 R14: 000000c03e2b4340 R15: 0000000000000000
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Mem-Info:
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: active_anon:1919323 inactive_anon:1905 isolated_anon:0
active_file:172 inactive_file:73 isolated_file:0
unevictable:4617 dirty:4 writeback:0 unstable:0
slab_reclaimable:35806 slab_unreclaimable:25291
mapped:526606 shmem:524808 pagetables:9410 bounce:0
free:25658 free_pcp:1747 free_cma:0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 active_anon:7677292kB inactive_anon:7620kB active_file:688kB inactive_file:292kB unevictable:18468kB isolated(anon):0kB isolated(file):0kB mapped:2106424kB dirty:16kB writeback:0kB shmem:2099232kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1214464kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: lowmem_reserve[]: 0 2965 7907 7907 7907
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA32 free:44580kB min:25292kB low:31612kB high:37932kB active_anon:2954028kB inactive_anon:1292kB active_file:0kB inactive_file:888kB unevictable:0kB writepending:8kB present:3129172kB managed:3063636kB mlocked:0kB kernel_stack:432kB pagetables:11840kB bounce:0kB free_pcp:3164kB local_pcp:728kB free_cma:0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: lowmem_reserve[]: 0 0 4942 4942 4942
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 Normal free:42144kB min:42156kB low:52692kB high:63228kB active_anon:4723264kB inactive_anon:6328kB active_file:1324kB inactive_file:500kB unevictable:18468kB writepending:8kB present:5242880kB managed:5068608kB mlocked:18468kB kernel_stack:4784kB pagetables:25800kB bounce:0kB free_pcp:3824kB local_pcp:976kB free_cma:0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: lowmem_reserve[]: 0 0 0 0 0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 DMA32: 554*4kB (UME) 483*8kB (UME) 831*16kB (UME) 318*32kB (UME) 129*64kB (UME) 33*128kB (UE) 3*256kB (UM) 0*512kB 2*1024kB (UM) 0*2048kB 0*4096kB = 44848kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 Normal: 1590*4kB (UMEH) 1296*8kB (UMEH) 955*16kB (UMEH) 296*32kB (UMEH) 7*64kB (MH) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42312kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 527191 total pagecache pages
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages in swap cache
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Free swap = 0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Total swap = 0kB
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 2097011 pages RAM
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages HighMem/MovableOnly
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 59973 pages reserved
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages cma reserved
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: 0 pages hwpoisoned
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Tasks state (memory values in pages):
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 378] 0 378 86394 867 663552 0 -250 systemd-journal
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 536] 0 536 70033 4486 94208 0 -1000 multipathd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 643] 102 643 22718 531 81920 0 0 systemd-timesyn
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 679] 100 679 4796 730 77824 0 0 systemd-network
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 698] 101 698 6207 1775 90112 0 0 systemd-resolve
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 722] 0 722 5040 746 61440 0 -1000 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 827] 0 827 61160 1228 106496 0 0 accounts-daemon
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 835] 0 835 2134 560 53248 0 0 cron
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 836] 103 836 2003 991 61440 0 -900 dbus-daemon
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 842] 0 842 324985 236 172032 0 -900 droplet-agent
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 847] 0 847 20481 761 57344 0 0 irqbalance
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 849] 0 849 7469 2819 94208 0 0 networkd-dispat
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 852] 0 852 59077 794 106496 0 0 polkitd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 859] 104 859 56086 526 86016 0 0 rsyslogd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 865] 0 865 292647 4440 303104 0 -900 snapd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 868] 0 868 4395 1039 77824 0 0 systemd-logind
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 871] 0 871 98871 525 131072 0 0 udisksd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 872] 0 872 948 504 45056 0 0 atd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 873] 0 873 622 143 45056 0 0 none
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 874] 0 874 375917 4672 311296 0 -999 containerd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 888] 0 888 1838 402 53248 0 0 agetty
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 911] 0 911 79705 516 126976 0 0 ModemManager
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 912] 0 912 1457 387 45056 0 0 agetty
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 917] 0 917 652 118 45056 0 0 sshguard
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 919] 0 919 652 24 45056 0 0 sshguard
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 920] 0 920 1034 330 45056 0 0 sshg-parser
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 921] 0 921 19538 806 53248 0 0 sshg-blocker
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 922] 0 922 652 24 45056 0 0 sshguard
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 923] 0 923 652 443 45056 0 0 sshg-fw-iptable
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 924] 0 924 159700 971 741376 0 0 journalctl
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 942] 0 942 27031 2689 102400 0 0 unattended-upgr
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 948] 0 948 3043 729 61440 0 -1000 sshd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 553186] 0 553186 361527 5005 356352 0 -500 dockerd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631350] 1002 631350 4762 823 69632 0 0 systemd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631357] 1002 631357 43209 1272 110592 0 0 (sd-pam)
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631443] 1002 631443 2219 455 49152 0 0 screen
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 631444] 1002 631444 2606 999 57344 0 0 bash
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 633355] 1002 633355 195674 22178 282624 0 0 metrics-collect
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 664318] 0 664318 750436 17304 2252800 0 0 grafana-agent
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670974] 112 670974 543687 15391 262144 0 -900 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670976] 112 670976 543938 520398 4308992 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670977] 112 670977 543720 516982 4304896 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670978] 112 670978 543687 4689 180224 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670979] 112 670979 543855 865 180224 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670980] 112 670980 18189 615 139264 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670981] 112 670981 543794 781 172032 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670982] 112 670982 543792 776 159744 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 670983] 112 670983 544464 2378 204800 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671211] 1020 671211 183655 2671 159744 0 0 visor
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671236] 1020 671236 784480 495752 5017600 0 0 vega
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671237] 1020 671237 1107523 811623 7364608 0 0 vega
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671310] 995 671310 188400 4246 200704 0 0 caddy
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671387] 112 671387 544345 7955 290816 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671406] 112 671406 546181 525444 4345856 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671434] 112 671434 544577 514269 4337664 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 671506] 112 671506 565894 10351 462848 0 0 postgres
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743022] 0 743022 5040 684 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743023] 0 743023 5040 684 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743024] 0 743024 5040 684 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743025] 0 743025 5040 661 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743026] 0 743026 5040 661 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743027] 0 743027 5040 477 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743028] 0 743028 5040 661 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743029] 0 743029 5040 661 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743030] 0 743030 5040 658 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743033] 0 743033 5040 668 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743034] 0 743034 5040 668 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743035] 0 743035 5040 668 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743039] 0 743039 5040 758 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743040] 0 743040 5040 758 57344 0 0 systemd-udevd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743047] 0 743047 3043 821 61440 0 0 sshd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: [ 743049] 109 743049 3043 563 57344 0 0 sshd
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/vegavisor.service,task=vega,pid=671237,uid=1020
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Out of memory: Killed process 671237 (vega) total-vm:4430092kB, anon-rss:3246492kB, file-rss:0kB, shmem-rss:0kB, UID:1020 pgtables:7192kB oom_score_adj:0
Grafana charts
Analysis
I have analyzed it, and here are my findings:
Memory fragmentation
Dec 21 00:43:18 n00.testnet.vega.xyz kernel: Node 0 Normal: 1590*4kB (UMEH) 1296*8kB (UMEH) 955*16kB (UMEH) 296*32kB (UMEH) 7*64kB (MH) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42312kB
This line looks fine because we still have some blocks of continuous memory, but the amount of free memory is around 42Mb.
Free memory in the Normal space
Node 0 Normal free:42144kB min:42156kB low:52692kB high:63228kB active_anon:4723264kB inactive_anon:6328kB active_file:1324kB inactive_file:500kB unevictable:18468kB writepending:8kB present:5242880kB managed:5068608kB mlocked:18468kB kernel_stack:4784kB pagetables:25800kB bounce:0kB free_pcp:3824kB local_pcp:976kB free_cma:0kB
I understand that the OOM killer has been invoked because the free memory(42144kB) fell below the min threshold(42156kB).
Processes table dump
Then I switched to the memory utilized by processes and I wrote simple script:
#!/bin/python
import re
results = []
page_size_bytes = 4096
idx_total_vm = 9
idx_rss = 10
idx_name = -2
gigabyte = 1024**3 # bytes
with open("/home/daniel/tmp/oom-dump-21-12-2022.txt") as file:
for line in file.readlines():
line_parts = re.split('\s+', line)
try:
results.append((line_parts[idx_name], int(line_parts[idx_rss])*page_size_bytes, int(line_parts[idx_total_vm])*page_size_bytes))
except:
continue
results.sort(key= lambda item: item[1], reverse=True)
print("%15s : %9s, %9s" % ("process name", "rss", "vsz"))
for row in results:
print("%15s : %3f GB , %3f GB" % (row[0], row[1]/gigabyte, row[2]/gigabyte))
total_rss = sum([row[1] for row in results])
total_vsz = sum([row[2] for row in results])
print("%15s : %3f GB , %3f GB" % ("total", total_rss/gigabyte, total_vsz/gigabyte))
It gives me the following results:
process name : rss, vsz
vega : 3.096096 GB , 4.224865 GB
postgres : 2.004410 GB , 2.083515 GB
postgres : 1.985161 GB , 2.074959 GB
postgres : 1.972130 GB , 2.074127 GB
postgres : 1.961781 GB , 2.077396 GB
vega : 1.891144 GB , 2.992554 GB
metrics-collect : 0.084602 GB , 0.746437 GB
grafana-agent : 0.066010 GB , 2.862686 GB
postgres : 0.058712 GB , 2.074001 GB
postgres : 0.039486 GB , 2.158714 GB
postgres : 0.030346 GB , 2.076511 GB
dockerd : 0.019093 GB , 1.379116 GB
postgres : 0.017887 GB , 2.074001 GB
containerd : 0.017822 GB , 1.434010 GB
multipathd : 0.017113 GB , 0.267155 GB
snapd : 0.016937 GB , 1.116360 GB
caddy : 0.016197 GB , 0.718689 GB
networkd-dispat : 0.010754 GB , 0.028492 GB
unattended-upgr : 0.010258 GB , 0.103115 GB
visor : 0.010189 GB , 0.700588 GB
postgres : 0.009071 GB , 2.076965 GB
systemd-resolve : 0.006771 GB , 0.023678 GB
(sd-pam) : 0.004852 GB , 0.164829 GB
accounts-daemon : 0.004684 GB , 0.233307 GB
systemd-logind : 0.003963 GB , 0.016766 GB
bash : 0.003811 GB , 0.009941 GB
dbus-daemon : 0.003780 GB , 0.007641 GB
journalctl : 0.003704 GB , 0.609207 GB
systemd-journal : 0.003307 GB , 0.329567 GB
postgres : 0.003300 GB , 2.074642 GB
systemd : 0.003139 GB , 0.018166 GB
sshd : 0.003132 GB , 0.011608 GB
sshg-blocker : 0.003075 GB , 0.074532 GB
polkitd : 0.003029 GB , 0.225361 GB
postgres : 0.002979 GB , 2.074409 GB
postgres : 0.002960 GB , 2.074402 GB
irqbalance : 0.002903 GB , 0.078129 GB
systemd-udevd : 0.002892 GB , 0.019226 GB
systemd-udevd : 0.002892 GB , 0.019226 GB
systemd-udevd : 0.002846 GB , 0.019226 GB
systemd-network : 0.002785 GB , 0.018295 GB
sshd : 0.002781 GB , 0.011608 GB
systemd-udevd : 0.002609 GB , 0.019226 GB
systemd-udevd : 0.002609 GB , 0.019226 GB
systemd-udevd : 0.002609 GB , 0.019226 GB
systemd-udevd : 0.002548 GB , 0.019226 GB
systemd-udevd : 0.002548 GB , 0.019226 GB
systemd-udevd : 0.002548 GB , 0.019226 GB
systemd-udevd : 0.002522 GB , 0.019226 GB
systemd-udevd : 0.002522 GB , 0.019226 GB
systemd-udevd : 0.002522 GB , 0.019226 GB
systemd-udevd : 0.002522 GB , 0.019226 GB
systemd-udevd : 0.002510 GB , 0.019226 GB
postgres : 0.002346 GB , 0.069386 GB
0 : 0.002148 GB , 0.011608 GB
cron : 0.002136 GB , 0.008141 GB
systemd-timesyn : 0.002026 GB , 0.086662 GB
rsyslogd : 0.002007 GB , 0.213951 GB
udisksd : 0.002003 GB , 0.377163 GB
ModemManager : 0.001968 GB , 0.304050 GB
atd : 0.001923 GB , 0.003616 GB
systemd-udevd : 0.001820 GB , 0.019226 GB
screen : 0.001736 GB , 0.008465 GB
sshg-fw-iptable : 0.001690 GB , 0.002487 GB
agetty : 0.001534 GB , 0.007011 GB
agetty : 0.001476 GB , 0.005558 GB
sshg-parser : 0.001259 GB , 0.003944 GB
droplet-agent : 0.000900 GB , 1.239719 GB
none : 0.000546 GB , 0.002373 GB
sshguard : 0.000450 GB , 0.002487 GB
sshguard : 0.000092 GB , 0.002487 GB
sshguard : 0.000092 GB , 0.002487 GB
total : 13.465000 GB , 46.040333 GB
Questions
My question no 1 is:
Why can I see total memory usage(the RSS) of 13.46GB if I have 8GB on the system?
Does the RSS include shared libraries/memory in it?
So let's say I have 2GB shared memory for PostgreSQL; then my usage will be about 8GB. Is it correct?
My question no 2 is:
I have a gap in the grafana because I collect metrics every 60 seconds, so It does not show me spikes unless the spike is when the Prometheus scrapes metrics.
Do you know some exporter or tool that exports Prometheus metrics of memory allocation as a counter? So each memory allocation increases the value of the metric, then I can compute the rate of the allocation or increase value every given time period?
Or maybe you can recommend me something else to show those abnormal allocations with my monitoring gap?
