Score:1

What does the MEMORY(bytes) value mean in "kubectl top nodes" on Windows AKS nodes?

in flag

When I run kubectl top nodes on my Azure Kubernetes Services cluster with Windows nodes, I get a value back for MEMORY(bytes):

PS >kubectl top nodes
NAME                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
...
akswinxxxxxxxxx                     163m         8%     2407Mi          55%
...

This number (2407Mi) is not the Allocated value as returned by kubectl describe node, which (when converted to the same unit) is 2084Mi:

PS > kubectl describe node akswinxxxxxxxxx
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests         Limits
  --------           --------         ------
  cpu                1070m (56%)      500m (26%)
  memory             2134435Ki (48%)  2412Mi (56%)
  ephemeral-storage  0 (0%)           0 (0%)

(I've been seeing much larger differences in other situations.)

It's also not the same as the sum of the working sets (6398Mi) or private memory (3822Mi) of the processes running on the node:

PS C:\C\6428530b2b53f975bb582acc2f12458ca138075069d33d663f3b9e2ec70edbaf> (get-process |Measure-Object WorkingSet64 -Sum).Sum / 1024 / 1024
6397.7109375
PS C:\C\6428530b2b53f975bb582acc2f12458ca138075069d33d663f3b9e2ec70edbaf> (get-process |Measure-Object PrivateMemorySize64 -Sum).Sum / 1024 / 1024
3822.328125

It's not the sum of the Pod memory either, as reported by crictl stats (1531Mi):

PS >crictl stats
CONTAINER           CPU %               MEM                 DISK                INODES
2bdfd79f404af       0.00                33.39MB             71.3MB              0
32ea5bc86bacf       0.00                34.36MB             37.75MB             0
511eb449bc0ae       0.00                37.55MB             71.3MB              0
52a1bd98c3558       3.09                222MB               147.8MB             0
6428530b2b53f       0.00                73.54MB             37.75MB             0
6bd5050e8bb4c       0.00                37.68MB             37.75MB             0
78a51582874ec       0.00                241.5MB             104.9MB             0
bb1224de3c87e       0.00                34.14MB             71.3MB              0
e19566f08ca1e       0.00                38.27MB             37.75MB             0
e633436686347       0.00                329.5MB             172MB               0
f82c4306b4692       0.00                34.79MB             37.75MB             0
fa36b455d852f       0.00                235.8MB             104.9MB             0
fe7d9f0193f17       0.00                178MB               306.2MB             0

It also doesn't match "total memory minus free physical memory" on the node (5101Mi):

PS >$OS =  Get-WmiObject -Class WIN32_OperatingSystem
PS> ($OS.TotalVisibleMemorySize - $OS.FreePhysicalMemory)/1024
5101.25390625

As indicated in the Kubernetes docs, the value is meant to indicate a node's Working Set. So, what does that mean and how is it measured? How can I use it to monitor a node's health or available resources?

(Note: On the Internet, I often find references to Kubernetes's metrics-server. However, that does not appear to be running on my AKS Windows nodes.)

Update 2022-12-05: I know that the number is meant to indicate, in general, how much of the node's memory is currently being used. However, I really want to know more details about that. As it doesn't match the Pod memory numbers, it seems to include system processes, right? So, how can it be compared to Allocatable (the MEMORY% percentage), which is what is available for Pods? And why doesn't it match the numbers reported directly on the node (see above)? In a nutshell, I'd like to know how the number is actually calculated - on Windows.

Fabian Schmied avatar
in flag
I've now posted this question as an AKS Github issue as well: https://github.com/Azure/AKS/issues/3390. I'll document the answer here if I get one.
Score:0
ru flag

The "MEMORY (bytes)" value shown by kubectl top nodes isn't the same as the "Allocated" value that is shown by kubectl describe <node>. The "Allocated" value indicates the amount of memory that has been allocated to the node by the Kubernetes scheduler, based on the memory requests and limits that have been set for the pods running on the node. The "MEMORY (bytes)" value, on the other hand, indicates the actual amount of memory that is being used by the processes running on the node at the time the kubectl top nodes command is run.

You can use the "MEMORY (bytes)" value shown by kubectl top nodes to monitor the health of your nodes and to ensure that they have sufficient memory available to run their workloads. If the value is consistently close to the "Allocated" value, it may indicate that the node is running low on memory and may need to be resized or have its resources reallocated.

Fabian Schmied avatar
in flag
Thanks for the answer! "actual amount of memory that is being used by the processes running on the node at the time the kubectl top nodes command is run" -> I'd like to know more details about that. The number reported does not match the sum of the memory used by the processes on the node (see my question). It seems to include system processes, so can it really be compared to the Pod memory requests (_Allocated_)? Does it include the memory reserved by AKS for the system and kubelet? If not, how can I use it for predicting whether the nodes still have enough memory for workloads?
Fabian Schmied avatar
in flag
(I'll clarify my question accordingly.)
TheTanadu avatar
ru flag
Looks like you're on to something. I didn't go into the operation of the k8 so thoroughly, my answer was heavily based on the documentation, I must admit.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.