Disk spool

The disk spool is an extra delivery layer that turns “deliver or fail” into “deliver, eventually, through almost anything”. It is on by default: it’s the single most impactful reliability setting, so every run gets it unless you opt out.

The default spool

With disk left at its default, the spool lives under ~/.metrana/spools/<hostname> (override the base with METRANA_DISK_SPOOL_DIR; see Where the spool lives for the hostname component, and read it before running on Slurm), capped at 8 GiB per run. The default is best-effort: at init() the engine creates the directory, writes a probe file and takes an exclusive lock on this process’s writer directory. If any of that fails — a read-only home, no space — init() warns and continues without a spool rather than failing your job. Read that warning: it means a sustained outage can no longer fail over to disk. The one exception is init_mode="offline", which cannot run without a spool and raises MetranaSpoolUnavailableError instead.

Requiring one, or turning it off

An explicit path (or DiskConfig) is a requirement: init() raises MetranaSpoolUnavailableError if the spool cannot be used. Size it for the disk it lives on:

metrana.init(..., disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=16 << 30))
# a bare path also works, but takes the 8 GiB default cap — prefer an explicit size:
metrana.init(..., disk="/var/spool/metrana")
# opt out entirely (no failover, no warning):
metrana.init(..., disk=None)

How eagerly the spool engages is a profile choice: patient (block, spool only on real outages) or bounded-latency (spool_then_raise, spool on any sustained pressure).

Each run gets its own sub-directory under that path — the spool actually lives at <dir>/<workspace>/<project>/<run_name> — so many runs can safely share one base disk= directory without ever mixing segments.

How it works

While gRPC is healthy, data goes straight over the wire — the spool directory holds no points, only the run’s step cursors: a small snapshot refreshed about once a minute plus an append-only journal of what changed since. That is what lets a hard kill resume where it stopped, and what a later process reads to continue your steps. When delivery gives up (the gRPC retry budget is spent), the logger fails over to the spool — every subsequent point is fsync’d to disk instead of dropped — and a background recovery channel keeps trying to re-establish gRPC. Once the wire returns, it drains the spool to the backend, in order, switches delivery back to gRPC, and reclaims the disk as it goes.

gRPC healthy:   log ──▶ gRPC ──▶ backend
gRPC down:      log ──▶ disk spool (fsync)     ◀── recovery channel retries gRPC
gRPC recovers:  disk spool ──drain (in order)──▶ backend, then switch back to gRPC

Because the spool sits behind the delivery decision, it protects against every backend-side failure — not just network blips, but ones retries alone can never fix: an expired token, an accidentally revoked authorization, an unauthenticated rejection, a backend refusing requests, a regional outage. The data waits on disk and the recovery channel retries reconnecting indefinitely, so the moment the problem is fixed (token rotated, permissions restored) the backlog ships automatically. Nothing is dropped while there is room on disk (bounded by max_size_bytes).

Durable across restarts

The spool is durable across process restarts. If a process dies while data is spooled and the next process starts with the same disk= directory, it resumes from the leftover spool and keeps delivering — no manual replay needed for the crash-and-restart case.

Starting without the backend

A restarted process keeps going without the backend when the previous process left data on the spool: it resumes straight from that leftover spool, skipping the backend handshake.

A run can also start without the backend — a brand-new run on an air-gapped cluster, or an init() whose handshake fails. That is a separate mode with its own delivery step; see Offline logging.

Replaying a leftover spool

If the outage outlasts the run — you close() (or the process ends) while data is still stranded — Metrana warns, and the CloseInfo reports has_undelivered_spool. The data is safe on disk; deliver it later with replay_spool:

import metrana

# Re-deliver a leftover spool to its run. No prior init() needed; the identity args must
# match the run that produced the spool.
info = metrana.replay_spool(
    "/var/spool/metrana",            # the BASE disk path, not the per-run sub-directory
    workspace_name="my-team",
    project_name="my-project",
    run_name="run-2026-06-28",
    api_key="...",                   # or via METRANA_API_KEY
)
print(info.metrics.float_points_sent, "points re-delivered")

replay_spool re-derives the per-run sub-directory from the identity arguments, exactly as the run wrote it — so pass the base disk= path, not the <workspace>/<project>/<run_name> sub-directory. Omit the path for a run that used the default spool and is replayed from the node that wrote it (the default is <METRANA_DISK_SPOOL_DIR or ~/.metrana/spools>/<hostname>; from another node pass that path with the writing node’s hostname). If the spool has no close marker (the original process crashed mid-run), replay leaves the run open so it can still be resumed. A spool written by an offline run is initialised on the backend first, then delivered.

Multiple processes logging to one run

Nothing to configure. Every logger process gets its own writer directory under the run’s spool directory (<run dir>/w-<created ms>-<logger_id>-<token>/, so names sort by creation time), so distributed ranks on one host, a forked child, or a restarted process all spool the same run with the same disk= setting and never share a file. Each live process holds an exclusive lock on its writer directory from before its first byte until it exits; the kernel drops the lock with the process, so a directory that holds data with a free lock belongs to a process that is gone. That is the whole protocol — no PIDs, no heartbeats — and it drives two things:

• Leftovers are delivered first. A process that starts online and finds earlier processes’ spools under the run (its own previous incarnation, a crashed sibling) delivers them before anything of its own: it starts on its spool, its recovery channel replays the leftovers oldest first under their own writer identities, then drains its own spool and returns to gRPC. The server never sees a series’ steps out of order across a restart chain. If the backend is unreachable at that start, the leftovers still identify the run, and the process continues on its spool as during any outage.
• Cursors carry over. Every leftover’s metadata is folded into the new process’s, so auto-assigned steps and episodes continue where the earlier process stopped — no logger_id pinning required.
• Offline loggers cannot deliver anything: they report leftovers at close (CloseInfo.foreign_spools), and one replay_spool of the run delivers every writer, oldest first, honouring only the newest spool’s close marker.

os.fork() needs nothing special: the child’s first log call rebuilds its logger with its own writer directory, offline runs included. Forked DataLoader / multiprocessing workers still shouldn’t be logging metrics; the main process is the logger.

Python 3.12+ prints a DeprecationWarning here (“use of fork() may lead to deadlocks in the child”), and gRPC logs that it skipped its fork handlers. Both are generic warnings about forking a multi-threaded process, not signs of a problem: Metrana holds no lock across the fork, and the child builds its own connection rather than using the inherited one. See Distributed logging.

Where the spool lives

The writer lock is a flock, and a flock only means something within one machine. Over a network filesystem it may be emulated, downgraded to local-only by a mount option (-o nolock, local_lock=, Lustre’s localflock), or quietly hand two hosts the same lock — and nothing in userspace can tell you which you got. The hostname in the default path is what stops that mattering: everything beneath it belongs to one kernel, so “data on disk plus a free lock = a dead writer” stays true, and no node can sweep, replay, consume or delete a spool another node is still writing.

The same limit applies to replay_spool: it takes the writer’s lock before reading and refuses a spool a logger is still writing, so a replay can never delete segments underneath a live process — on that machine. Two nodes pointed at one network filesystem are outside what the lock can promise, which is the reason for the namespacing rather than an oversight. If your setup shares a filesystem between machines, give disk= a path that is unique per node (below) rather than relying on the default.

The cost is that a restart under a new hostname — a rescheduled Deployment pod, a Slurm job requeued onto other nodes — cannot see what its predecessor left. What happens then depends on whether that spool still owes anything:

• Nothing owed → its step cursors carry over. A spool whose data all reached the backend still records where its steps got to, and a logger on another node reads that, so auto-assigned steps and episodes continue instead of restarting at zero. This matters most for offline runs, which have no handshake to seed them.

• Data still owed → init() refuses. Continuing would set this process’s next step above data that has not arrived yet, and the ingestion service rejects a step below the last one it accepted — so replaying that spool afterwards would drop it, not merely deliver it late. Rather than lose it quietly, init fails and names the directory.

  Which fix applies depends on whose spool it is:

  • Your own run, restarting under a new hostname. Point disk= at that directory — the one carrying the old hostname. Its spool becomes this process’s own leftover, so the ordinary recovery path delivers it before any new data, in order, with no manual step. The two must share an orchestration id for that: framework job ids (SLURM_JOB_ID, TORCHELASTIC_RUN_ID) survive a requeue, otherwise set METRANA_ORCHESTRATION_ID or pass resume_strategy="allow".
  • A different process that is still running. Give this one a disk= path of its own and leave that spool to its owner.

  Replaying by hand with replay_spool also works, and is the right move when you only want the data delivered and are not continuing the run.

Shared filesystem + changing hostname: set disk= yourself

If both are true of your setup — the spool base is reachable from several machines, and the hostname changes across restarts — the default is not merely suboptimal, it will stop your restarts. Every interrupted run leaves a spool the successor can neither deliver nor order itself behind, so init() raises MetranaUndeliverableSpoolError until you replay that directory by hand. That is the safe answer, but it is not one you want in a requeue loop.

Set disk= to a path that identifies the worker rather than the machine, and the whole problem goes away: the successor lands on its own predecessor’s directory, where the writer lock applies, so leftovers are delivered before its own data automatically and no replay is needed.

init_mode="offline" users tend to arrive here already — offline cannot run without a spool and raises rather than degrading, so they have had to think about disk — but an online run takes the default silently and only meets the constraint when a restart refuses.

Slurm: key the path by node id

$SLURM_NODEID is the node’s index within the allocation, so it survives a requeue even when the physical machine does not:

import os
metrana.init(..., disk=f"/scratch/{os.environ['USER']}/metrana/{os.environ['SLURM_NODEID']}")

An explicit disk= path is used verbatim — no hostname is appended — so a requeued rank finds its own predecessor directly and gets the whole recovery path back: leftovers delivered before new data, cursors continued, directories cleaned up, no manual replay. It swaps one assumption for the lock: that the scheduler really did kill the previous allocation before starting the replacement. Schedulers do; a partitioned straggler would not.

On a Slurm cluster with an NFS home — where ~/.metrana/spools is shared and requeues move you between nodes — treat this as the required setup rather than a tuning option.

If you would rather not rely on that, point disk= at node-local scratch ($TMPDIR, $SLURM_TMPDIR) — nothing is shared, so the lock is authoritative again, but the spool dies with the node, which is the persistence it exists to give you.

Kubernetes

The choice is usually made for you. A StatefulSet has stable pod names (worker-0), so the default is fine. A Deployment or Job with a PVC is the one combination where the pod name churns under a volume that outlives it — give those a stable disk= path (the rank, the ordinal, anything that identifies the worker rather than the machine). If the volume is node-local and dies with the pod, none of this applies: there is nothing to strand.

Capacity

The spool is bounded by max_size_bytes, which defaults to 8 GiB per run. Two things make that go a long way: segments are zstd-compressed by default, so 8 GiB holds far more than 8 GiB of raw points, and the budget is reclaimed continuously as the recovery channel drains delivered segments.

Offline runs: the cap is the whole run, not a rolling window

Reclamation needs the recovery channel, which needs the network. An offline run never has one, so nothing is freed until you replay — the 8 GiB is the total budget for everything the run will ever log, and hitting it stops the run under the delivery policy.

Roughly, spooled points cost ~3 bytes each (~9 bytes on the wire, ~3× from zstd), so 8 GiB is a few billion points. For ordinary ML logging that is years; for high-fanout RL it is not long — 1024 envs × 16 metrics × 100 minor points once a minute is ~27K points/s, which fills the default in about 30 hours. Size max_size_bytes for the run’s full length before starting one air-gapped.

Still, size it for your runs — the default can surprise in either direction:

• A long, high-rate run during a lengthy outage can fill 8 GiB, after which new data hits the delivery policy (all_or_crash by default). If you have the disk, be generous — a bigger spool buys a longer survivable outage:

  metrana.init(..., disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=64 << 30))  # 64 GiB

• Many small runs sharing one disk can collectively surprise you with how much space they hold. If your runs are small, restrict it so a forgotten spool can’t eat the volume:

  metrana.init(..., disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=1 << 30))   # 1 GiB

There is no special “unlimited” sentinel — max_size_bytes is a byte count. To effectively uncap it, set it to whatever your disk can hold (a very large value); the spool then only stops when the filesystem does.

Configuration

DiskConfig flattens all spool tuning into one constructor. The defaults match the underlying engine:

Field	Default	Purpose
dir (positional)	(required)	Base spool directory.
max_size_bytes	8 << 30 (8 GiB)	Spool size cap; reclaimed as segments drain.
segment_size_bytes	100 << 20 (100 MiB)	Size of each on-disk segment file.
compression	DiskCompression.Zstd	On-disk compression (Zstd or none).
zstd_level	3	zstd compression level.
io_max_retries	60	Disk I/O retries before giving up; None retries forever.

metrana.init(..., disk=metrana.DiskConfig(
    "/var/spool/metrana",
    max_size_bytes=32 << 30,
    compression=metrana.DiskCompression.Zstd,
))

Passing disk="/path" (a bare string) uses all of these defaults. See the configuration reference.