How can I set up interactive-job-only or batch-job-only partition on a SLURM cluster?

wdg

5/21/23, 5:44 AM

I'm managing a PBS/torque HPC cluster, and now I'm setting up another cluster with SLURM. On the PBS cluster, I can set a queue to accept only interactive jobs by qmgr -c "set queue interactive_q disallowed_types = batch" and to accept only batch jobs by qmgr -c "set queue cu disallowed_types = interactive".

However, I cannot find the equivalent setups with SLURM after browsing through the official SLURM documents.

How can I set up a partition on SLURM cluster to only accept interactive or batch jobs?

195

0 + 0

hpc

torque

pbs

slurm

Score:3

Server

user10489

5/21/23, 6:33 AM

As of Slurm 20, there isn't any direct way to separate interactive jobs from batch jobs in a partition.

However, it is possible to use a job submit plugin which can distinguish the two in several ways; batch jobs have a job script (job_desc.script in Lua) associated with them where interactive jobs don't. Some interactive jobs have a PTY. I think there also may be a flag set that is new in Slurm 20.

Once the job submit plugin has identified a job as interactive or batch, it can either reject it for the given partition, or it can possibly change the partition it is being sent to, or it could modify the job in some other appropriate way (like assigning an "interactive" license to it or change the QOS) before final submission.

If you modify the job, then additional Slurm configuration can affect the job based on that.

0 + 0

wdg

5/21/23, 9:17 AM

This is indeed a good idea. Job submitted by `sbatch` will have its field `BatchFlag` set to 1. This is great, but according to https://github.com/SchedMD/slurm/blob/master/src/plugins/job_submit/lua/job_submit_lua.c, it does not seem that the lua plugin can access `BatchFlag` field. Writing C plugin is too troublesome, as the compilation of SLURM is done by the vendor, not me. Right now, I can only tell the job is interactive if `job_desc.argv[job_desc.argc-1]=="bash"`. This is certainly not sufficient.

user10489

5/21/23, 12:25 PM

I wrote my lua job submit plugin for slurm 19 or earlier, and checking for the script works fine, so the batchflag field is redundant anyway. Maybe they'll add it to a future version of the lua plugin. Look at `job_desc.script`

wdg

5/21/23, 2:28 PM

Did you mean that for interactive jobs, `job_desc.script` is emtpy, but for batch jobs, they are not empty?

user10489

5/22/23, 12:03 AM

Right. Interactive jobs have no script and `job_desc.script` will be empty / not set.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How can I set up interactive-job-only or batch-job-only partition on a SLURM cluster?

TH: ฉันจะตั้งค่าพาร์ติชันเฉพาะงานแบบโต้ตอบหรือแบบกลุ่มเฉพาะงานบนคลัสเตอร์ SLURM ได้อย่างไร

RO: Cum pot configura o partiție interactivă numai pentru lucrări sau pe lot pe un cluster SLURM?

RU: Как настроить раздел только для интерактивных или пакетных заданий в кластере SLURM?

VI: Làm cách nào tôi có thể thiết lập phân vùng chỉ công việc tương tác hoặc chỉ công việc hàng loạt trên cụm SLURM?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.