CPU Job Statistics
Slurm has to be configured to track job accounting data via the cgroup plug-in. This requires the following line in slurm.conf:
The above is in addition to the other usual cgroup-related plug-ins/settings:
Slurm will then create two top-level cgroup directories for each job, one for CPU utilization and one for CPU memory. Within each directory there will be subdirectories: step_extern, step_batch, step_0, step_1, and so on. Within these directories one finds task_0, task_1, and so on. These cgroups are scraped by a cgroup exporter. The table below lists all of the collected fields:
| Name | Description | Type |
|---|---|---|
cgroup_cpu_system_seconds |
Cumulative CPU system seconds for jobid | gauge |
cgroup_cpu_total_seconds |
Cumulative CPU total seconds for jobid | gauge |
cgroup_cpu_user_seconds |
Cumulative CPU user seconds for jobid | gauge |
cgroup_cpus |
Number of CPUs in the jobid | gauge |
cgroup_memory_cache_bytes |
Memory cache used in bytes | gauge |
cgroup_memory_fail_count |
Memory fail count | gauge |
cgroup_memory_rss_bytes |
Memory RSS used in bytes | gauge |
cgroup_memory_total_bytes |
Memory total given to jobid in bytes | gauge |
cgroup_memory_used_bytes |
Memory used in bytes | gauge |
cgroup_memsw_fail_count |
Swap fail count | gauge |
cgroup_memsw_total_bytes |
Swap total given to jobid in bytes | gauge |
cgroup_memsw_used_bytes |
Swap used in bytes | gauge |
cgroup_uid |
UID number of user running this job | gauge |
The cgroup exporter used here is based on the exporter by Trey Dock [1] with additional parsing of the jobid, steps, tasks and UID number. This produces an output that resembles (e.g., for system seconds):
Note that the UID of the owning user is stored as a gauge in cgroup_uid:
This is because accounting is job-oriented and having a UID of the user as a label would needlessly increase the cardinality of the data in Prometheus. All other fields are alike with jobid, step and task labels.
The totals for a job have an empty step and task, for example:
This is due to the organization of the cgroup hierarchy. Consider the directory:
Within this directory, one finds the following subdirectories:
job_247463/cpuacct.usage_user
job_247463/step_extern/cpuacct.usage_user
job_247463/step_extern/task_0/cpuacct.usage_user
This is the data most often retrieved and parsed for overall job efficiency which is why by default the cgroup_exporter does not parse step or task data. To collect all of it, add the --collect.fullslurm option. We run the cgroup_exporter with these options:
The --config.paths /slurm has to match the path used by Slurm under the top cgroup directory. This is usually a path that is something like /sys/fs/cgroup/memory/slurm.
Slurm 26.05 or newer
As of version 26.05 slurm will now use SLUID instead of job_JOBID in cgroupv2 directory structure. E.g.
To continue collecting data correctly you have two options. The preferred one would be to upgrade to version 26.05.2 or newer where you can set optionCgroupJobIdPaths=yesto revert back to using jobid's in cgroup directories - there is no downside to this choice.
Alternatively you can use the current version of cgroup exporter - version v0.3.1 or newer or source code as of June 4th, 2026 or newer. This version will collect SLUID as the jobid on these new versions of slurm. The current version of jobstats has also been changed to handle this change transparently.