When I run kubectl top nodes
on my Azure Kubernetes Services cluster with Windows nodes, I get a value back for MEMORY(bytes)
:
PS >kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
...
akswinxxxxxxxxx 163m 8% 2407Mi 55%
...
This number (2407Mi) is not the Allocated value as returned by kubectl describe node
, which (when converted to the same unit) is 2084Mi:
PS > kubectl describe node akswinxxxxxxxxx
...
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1070m (56%) 500m (26%)
memory 2134435Ki (48%) 2412Mi (56%)
ephemeral-storage 0 (0%) 0 (0%)
(I've been seeing much larger differences in other situations.)
It's also not the same as the sum of the working sets (6398Mi) or private memory (3822Mi) of the processes running on the node:
PS C:\C\6428530b2b53f975bb582acc2f12458ca138075069d33d663f3b9e2ec70edbaf> (get-process |Measure-Object WorkingSet64 -Sum).Sum / 1024 / 1024
6397.7109375
PS C:\C\6428530b2b53f975bb582acc2f12458ca138075069d33d663f3b9e2ec70edbaf> (get-process |Measure-Object PrivateMemorySize64 -Sum).Sum / 1024 / 1024
3822.328125
It's not the sum of the Pod memory either, as reported by crictl stats
(1531Mi):
PS >crictl stats
CONTAINER CPU % MEM DISK INODES
2bdfd79f404af 0.00 33.39MB 71.3MB 0
32ea5bc86bacf 0.00 34.36MB 37.75MB 0
511eb449bc0ae 0.00 37.55MB 71.3MB 0
52a1bd98c3558 3.09 222MB 147.8MB 0
6428530b2b53f 0.00 73.54MB 37.75MB 0
6bd5050e8bb4c 0.00 37.68MB 37.75MB 0
78a51582874ec 0.00 241.5MB 104.9MB 0
bb1224de3c87e 0.00 34.14MB 71.3MB 0
e19566f08ca1e 0.00 38.27MB 37.75MB 0
e633436686347 0.00 329.5MB 172MB 0
f82c4306b4692 0.00 34.79MB 37.75MB 0
fa36b455d852f 0.00 235.8MB 104.9MB 0
fe7d9f0193f17 0.00 178MB 306.2MB 0
It also doesn't match "total memory minus free physical memory" on the node (5101Mi):
PS >$OS = Get-WmiObject -Class WIN32_OperatingSystem
PS> ($OS.TotalVisibleMemorySize - $OS.FreePhysicalMemory)/1024
5101.25390625
As indicated in the Kubernetes docs, the value is meant to indicate a node's Working Set. So, what does that mean and how is it measured? How can I use it to monitor a node's health or available resources?
(Note: On the Internet, I often find references to Kubernetes's metrics-server. However, that does not appear to be running on my AKS Windows nodes.)
Update 2022-12-05: I know that the number is meant to indicate, in general, how much of the node's memory is currently being used. However, I really want to know more details about that. As it doesn't match the Pod memory numbers, it seems to include system processes, right? So, how can it be compared to Allocatable (the MEMORY%
percentage), which is what is available for Pods? And why doesn't it match the numbers reported directly on the node (see above)? In a nutshell, I'd like to know how the number is actually calculated - on Windows.