Restart systemd service when GPU memory limit is reached

BlackHawk

4/16/24, 10:30 AM

I'm using the following systemd service on Ubuntu 22.04 LTS. The service is saved in /lib/systemd/system/myService.service

[Unit]
Description=myService
RequiresMountsFor=/mnt/Data

[Service]
ExecStart=/usr/local/bin/project/main
Restart=always
RestartSec=5
StandardOutput=append:/usr/local/bin/project/stdout.log
StandardError=append:/usr/local/bin/project/stderr.log

[Install]
WantedBy=multi-user.target

My service is using the GPU. I would like to restart the service automatically if a certain amount of GPU memory usage (vram) is reached. I could use MemoryHigh and MemoryMax but I think this is for CPU memory only.

Is it possible to automatically restart a service when GPU memory is over a limit for a certain amount of time (e.g. 1min)? Peaks should not be detected but only when memory is too high for a longer amount of time.

Do you see any other improvements of the service? I would like that the service is always running, i.e. always restarting at boot and when it crashes. I think this I already achieved with the service file.

0 + 2

gpu

systemd

22.04

David

4/16/24, 12:52 PM

It may help to know why you want to do this.

BlackHawk

4/16/24, 4:41 PM

@David Because my service is running a Huggingface model which uses the GPU. Unfortunately, GPU memory usage increases over time. Thus, I would like to restart when a limit is reached.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Restart systemd service when GPU memory limit is reached

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.