How to monitor CPU and GPU usage amongst users?

Anthony Nash

2/26/23, 11:44 AM

I have several users (with user accounts as per /etc/passwd) who SSH onto an Ubuntu machine that I manage. This is my system info:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:    20.04
Codename:   focal

The HW is a two-socket Intel Xeon E3 (16 cores in total) with x2 NVidia GTX 970 GPU cards. There is approx 6 TB in the machine of internal HDD space.

Each user can use tmux to execute a process that persists after they've logged off. Please note, I don't have anything sophisticated such as a job manager like SLURM; I'm far from that so please don't suggest.

Could anyone recommend software to monitor the CPU and GPU usage by user and to report e.g., 1000 CPU/GPU hours etc., over a given period of time? The software must be able to record the User, CPU and GPU, and if possible the process/software executed.

144

0 + 0

users

software-sources

user-management

paladin

2/26/23, 1:06 PM

It's not a monitoring, but you may edit `/etc/security/limits.conf` and adjust user/group limits for CPU usage and other system ressources.

Score:1

Ubuntu

user10489

2/26/23, 12:34 PM

Something like sar and sadf (part of the sysstat package) can do full cpu accounting.

However there are currently few or no tools that can give good gpu accounting. Slurm can only do it by restricting and tightly scheduling gpu access, not by measuring actual use.

If one were to write a system like this, it would need to use the nvidia nvml libraries. The API for gpu montioring has completely changed several times in the last few years, so such a tool would require frequent rewrites to keep up with the changes in the nvidia driver and nvml library.

0 + 0

Anthony Nash

2/26/23, 1:09 PM

That you for your help. That's exactly what I needed to know. I'm happy with letting the GPU monitoring slip if it's going to be far too much work.

user10489

2/26/23, 1:11 PM

I use a solution that does sampling with collectd and a collectd python plugin available in pip. But this only does overall GPU usage, not per user usage.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How to monitor CPU and GPU usage amongst users?

TH: จะตรวจสอบการใช้งาน CPU และ GPU ของผู้ใช้ได้อย่างไร?

RO: Cum se monitorizează utilizarea CPU și GPU în rândul utilizatorilor?

RU: Как контролировать использование процессора и графического процессора среди пользователей?

VI: Làm cách nào để giám sát việc sử dụng CPU và GPU giữa những người dùng?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.