System metrics (autocapture)

Every run captures its hardware in the background: GPU utilization and memory, host CPU, memory and disk, and each process’s own footprint. You log nothing — init() starts a sampler thread and the series appear under the system/ prefix beside your training metrics.

metrana.init(organization="acme", workspace_name="research", project_name="llm", run_name="pretrain-7b")
# system/gpu/utilization, system/cpu/utilization, ... start arriving every 30s

Autocapture is on by default, samples every 30 seconds, and is Linux only. It never raises into your training loop: a missing driver, an unreadable counter or a rejected point means fewer series, never a failed run.

What is captured

Metric	Unit	Reported by
system/gpu/utilization	percent	The process that owns the GPU
system/gpu/memory_used_bytes	bytes	“
system/gpu/memory_utilization	percent	“
system/gpu/temperature_celsius	°C	“
system/gpu/power_watts	W	“
system/gpu/sm_clock_mhz	MHz	“
system/cpu/utilization	percent	One process per node
system/memory/used_bytes	bytes	“
system/memory/utilization	percent	“
system/disk/total_bytes	bytes	“
system/disk/free_bytes	bytes	“
system/disk/utilization	percent	“
system/process/rss_bytes	bytes	Every process, for itself
system/process/cpu_utilization	percent	“

A few of these mean something more specific than the name suggests:

• system/gpu/memory_utilization is how full the card is (used / total), not NVML’s utilization.memory, which measures how busy the memory bus was.
• system/memory/* reports the working set — usage minus reclaimable page cache — so a job that streams a large dataset doesn’t appear to sit at its limit all run.
• system/process/cpu_utilization is a percentage of one core, so a multi-threaded process legitimately exceeds 100%, the same convention top uses.
• system/disk/* describes the root filesystem, and utilization matches what df reports.

GPU metrics need the system extra

CPU, memory, disk and per-process metrics are read straight from /proc and /sys/fs/cgroup and need nothing extra. GPU metrics need NVIDIA’s NVML bindings:

pip install 'metrana[system]'

Without it, GPU series are skipped with a one-time warning and everything else still works.

Containers and cgroups

Inside a container, /proc/meminfo and os.cpu_count() describe the host, not your limits — a job capped at 16 GB on a 512 GB machine would read as 3% memory use right up until it is OOM-killed. Autocapture reads the cgroup instead, under both cgroup v1 and v2, and measures against the limit that actually applies to you.

Limits are inherited, so the tightest one in the hierarchy wins: a Slurm job slice or a Kubernetes pod parent commonly holds the cap while your own cgroup reads max. CPU utilization is scaled by the CPU quota where one is set, so a job throttled to two cores on a 128-core box reads as saturated rather than as 1.5% busy.

Several ranks on one node

Every rank of a distributed job runs its own sampler, but a node’s eight GPUs are visible to all eight ranks and its CPU and memory belong to none of them in particular. Rather than have you gate autocapture on rank, the processes of a run arbitrate between themselves so each piece of hardware is reported exactly once:

• Each GPU is reported by the rank that drives it, resolved from CUDA_VISIBLE_DEVICES, then LOCAL_RANK / SLURM_LOCALID / OMPI_COMM_WORLD_LOCAL_RANK, then an already-initialized Torch or JAX.
• Node metrics are reported by exactly one process per node, elected by lock.
• Process metrics need no arbitration — every process reports itself.

Arbitration is per-run, so two jobs sharing a node each report the hardware they hold and neither sees the other’s.

This works with torchrun, DDP, FSDP, HF Accelerate, DeepSpeed, Megatron, Slurm and JAX without configuration, because they all publish the same rank environment.

If you only call init() on rank 0

A run that initializes on one rank only would otherwise report a single GPU out of the node’s eight. The elected node reporter waits 45 seconds for slower ranks to claim their own devices, then takes over whatever is still unclaimed — so a rank-0-only run ends up charting every GPU it was allocated.

The 45-second grace means these extra series start a little after the run does. Ranks that start later than that keep quiet rather than fighting for a device already being reported, which keeps each series continuous.

Series identity

System series carry labels so you can separate them:

Label	On	Value
host	every system series	Hostname
gpu	GPU series	Driver index of the device
pid	process series only	Process id

They use the default ML_STEP scale, but their step is a sample counter, not your training step — step 40 of system/cpu/utilization and step 40 of loss are unrelated moments. Every point carries a wall-clock timestamp, so read these charts on the time axis, where they line up with each other and with your training metrics.

Configuration

init() argument	Environment variable	Default
capture_system_metrics	METRANA_CAPTURE_SYSTEM_METRICS	True
system_metrics_interval_secs	METRANA_SYSTEM_METRICS_INTERVAL_SECS	30.0

metrana.init(..., capture_system_metrics=False)          # off
metrana.init(..., system_metrics_interval_secs=10)       # sample every 10s

export METRANA_CAPTURE_SYSTEM_METRICS=false

Intervals below one second are raised to one: the CPU figures are deltas over the interval, and sub-second windows measure scheduling noise rather than load.

Turn it off if you already collect node metrics into Prometheus or Grafana and don’t want them twice — though what autocapture adds is per-run, per-rank data on the same timeline as your loss curve, which a node exporter can’t give you.

What is not captured

• MIG-partitioned GPUs are skipped. NVML reports no utilization for a MIG instance, and a partitioned card’s power and temperature belong to the whole device, which may be shared with another tenant. Non-partitioned GPUs on the same host are still captured.
• Non-NVIDIA GPUs. AMD/ROCm is not supported yet.
• macOS and Windows capture nothing; autocapture disables itself with a warning.

When a chart is missing

Each run records what autocapture found, so an empty chart is explainable after the fact. These appear as run attributes:

Attribute	Meaning
system/platform	Platform string of the machine
system/gpu/driver_version	NVIDIA driver version
system/gpu/visible_count	GPUs exposed to the process that created the run
system/gpu/host_device_count	GPUs on the node as a whole
system/gpu/mig_partitioned_count	How many were skipped for being MIG-partitioned
system/arbitration_mode	How processes agreed on ownership

visible_count against host_device_count is the quickest answer to “why am I only seeing four GPU series?” — usually because the job was allocated four of the node’s eight.

Failure handling

Autocapture is enabled without being asked for, so it is deliberately quiet and deliberately unobtrusive:

• Nothing it does raises into your training loop, and no failure closes the run.
• A counter the hardware doesn’t expose is dropped after one attempt; a transient error is retried and only given up on after five consecutive failures.
• Sampling stops for the run after five consecutive failed samples, with one warning.
• Warnings fire once, not once per sample.

See also

• Distributed logging — how several processes share one run.
• Steps, scales & labels — how a series is identified.
• Configuration — every init() argument.