Score:0

Server

Process with very high RAM usage takes very long to stop using force stop in Task Manager

rbaleksandar

4/13/24, 1:01 PM

We have a Windows Server 2016 with around physical 700GB RAM. A colleague of mine ran a machine learning script in Matlab, that loaded 25GB of data in RAM and during the training the RAM usage increased up to 350GB (usual behavior for many AI algorithms during training). This led to a big drop in performance for many other people (including the colleague who did this). He tried stopping it by force stoping the Matlab process tree (one node only) from the Task Manager.

The process is still "stopping" 2 hours later. We noticed that the RAM usage is gradually dropping but with around 200KB/s. Restarting the machine is currently not possible.

Any idea what is going on here? Normally killing a process should go past gentle shutdown procedures. At least this is my experience.

Update: a day later and the Matlab process has increased the RAM usage to 357GB

1 + 7

windows

memory

task-manager

process

windows-server-2016

Ramhound

4/13/24, 1:27 PM

“Normally killing a process should go past gentle shutdown procedures.” - The system has no memory to perform that operation since all physical and virtual memory was used by a single process. Seem strange it was able to use more than 5x the amount of memory that the system has.

paladin

4/13/24, 2:12 PM

It's Skynet emerging! Remove the power plug! ;-) It's probably a recursive problem. The machine learning process has created many sub-processes, which also create further sub-processes, killing them all at once is not possible, as the processes are unknown, so one after one process is being killed, while new processes are generated at the same time. Next time use Matlab within a VM and kill that VM.

Mokubai

4/13/24, 3:43 PM

Unless the machine actually had 350GB of physical RAM it probably wrote all of that used memory out to the page file and ballooned the page file out to 350+ GB as well. Now it is having to read in every page of memory that got paged out, invalidate it and then release it from the page table stored in memory. Whoever administrates this server should set some kind of sensible upper limit to page file size.

Ginnungagap

4/14/24, 6:28 AM

The question doesn't seem to have been edited and it clearly states that the server actually has 700GB of RAM so the script consumed about half of that, no reason for the pagefile to be involved (from what is stated in the question).

rbaleksandar

4/14/24, 8:18 AM

It's a server. It has a lot of memory (700GB physical). I will add the info to the question so that there is no confusion. Also one day later the memory usage is still 49-50%. :D

vidarlo

4/14/24, 8:52 AM

How was the process killed? `taskkill /f /im:foo.exe` will force it, and may be quicker. How much swap is used?

Greg Askew

4/14/24, 9:05 AM

The amount of memory is meaningless if it is fragmented. You would need a memory dump to analyze, but this is beyond your capability. `Restarting the machine is currently not possible`. Then use another server.

Score:0

Server

shodanshok

4/14/24, 11:07 AM

You probably killed the main process without killing the entire process tree. I suggest using ProcessExplorer to discover which processes are using RAM the most and then killing the entire process tree.

+ 2

rbaleksandar

4/14/24, 12:04 PM

Actually I went for the whole process tree (as stated in the question) even though in the tree there was just a single node. I did also try ProcessExplorer. The process currently cannot be stopped. Windows refuses to do so with the respective error message, which usually occurs whenever you try to stop something that is already in the process of stopping.

shodanshok

4/14/24, 1:16 PM

Can you share the precise error message (or a screenshot)?

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Process with very high RAM usage takes very long to stop using force stop in Task Manager

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.