Disk spool

The disk spool is an extra, encouraged delivery layer that turns “deliver or fail” into “deliver, eventually, through almost anything”. It’s off by default but strongly recommended for any real run — it’s the single most impactful reliability setting you can turn on.

Enabling it

Point init() at a directory:

metrana.init(..., disk="/var/spool/metrana")          # sensible defaults
# or tune it:
metrana.init(..., disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=16 << 30))

Each run gets its own sub-directory under that path — the spool actually lives at <dir>/<workspace>/<project>/<run_name> — so many runs can safely share one base disk= directory without ever mixing segments.

How it works

While gRPC is healthy, data goes straight over the wire and the spool is unused. When delivery gives up (the gRPC retry budget is spent), the logger fails over to the spool — every subsequent point is fsync’d to disk instead of dropped — and a background recovery channel keeps trying to re-establish gRPC. Once the wire returns, it drains the spool to the backend, in order, switches delivery back to gRPC, and reclaims the disk as it goes.

gRPC healthy:   log ──▶ gRPC ──▶ backend
gRPC down:      log ──▶ disk spool (fsync)     ◀── recovery channel retries gRPC
gRPC recovers:  disk spool ──drain (in order)──▶ backend, then switch back to gRPC

Because the spool sits behind the delivery decision, it protects against every backend-side failure — not just network blips, but ones retries alone can never fix: an expired token, an accidentally revoked authorization, an unauthenticated rejection, a backend refusing requests, a regional outage. The data waits on disk and the recovery channel retries reconnecting indefinitely, so the moment the problem is fixed (token rotated, permissions restored) the backlog ships automatically. Nothing is dropped while there is room on disk (bounded by max_size_bytes).

Durable across restarts

The spool is durable across process restarts. If a process dies while data is spooled and the next process starts with the same disk= directory, it resumes from the leftover spool and keeps delivering — no manual replay needed for the crash-and-restart case.

Offline startup

A restarted process can keep going without the backend — but only when the previous process left data on the spool (i.e. it was already failing over to disk when it stopped). It then resumes straight from that leftover spool, skipping the backend handshake.

Otherwise init() must reach the backend at startup and fails if it is unreachable. This covers a brand-new run, and also resuming a run that stopped while the backend was reachable (so the spool was empty/drained, leaving nothing on disk to resume from).

In short: the spool covers losing the backend during a run, including across a crash-and-restart, but not creating or resuming a run from a cold start with no prior spooled data. (Initialising a fresh run fully offline is a candidate for a future release.)

Replaying a leftover spool

If the outage outlasts the run — you close() (or the process ends) while data is still stranded — Metrana warns, and the CloseInfo reports has_undelivered_spool. The data is safe on disk; deliver it later with replay_spool:

import metrana

# Re-deliver a leftover spool to its run. No prior init() needed; the identity args must
# match the run that produced the spool, and the run must already exist on the backend.
info = metrana.replay_spool(
    "/var/spool/metrana",            # the BASE disk path, not the per-run sub-directory
    workspace_name="my-team",
    project_name="my-project",
    run_name="run-2026-06-28",
    api_key="...",                   # or via METRANA_API_KEY
)
print(info.metrics.float_points_sent, "points re-delivered")

replay_spool re-derives the per-run sub-directory from the identity arguments, exactly as the run wrote it — so pass the base disk= path, not the <workspace>/<project>/<run_name> sub-directory. If the spool has no close marker (the original process crashed mid-run), replay leaves the run open so it can still be resumed.

Multiple processes logging to one run

A single logging process needs no thought — run it however you like. For multiple processes writing the same run, isolate their spools before enabling disk:

Put each process in its own container with a per-pod volume (the ReadWriteOnce default) — each gets its own spool filesystem, and that isolation travels with your deployment config instead of a flag someone can forget.
Or give each a distinct base disk= path.

The only way to collide is to physically share one spool directory between concurrent writers (e.g. several ranks pointed at one network filesystem) — don’t.

fork() is handled for you: a forked child that logs gets a fresh logger with no spool (it logs straight to gRPC), so it can never collide with the parent’s spool — but it therefore won’t spool either. For a worker that needs durable spooling, start a fresh process (its own init()), not a fork. In practice a forked DataLoader / multiprocessing worker shouldn’t be logging metrics at all — the main process is the logger. See Distributed logging.

Capacity

The spool is bounded by max_size_bytes, which defaults to 8 GiB per run. Two things make that go a long way: segments are zstd-compressed by default, so 8 GiB holds far more than 8 GiB of raw points, and the budget is reclaimed continuously as the recovery channel drains delivered segments.

Still, size it for your runs — the default can surprise in either direction:

A long, high-rate run during a lengthy outage can fill 8 GiB, after which new data hits the delivery policy (all_or_crash by default). If you have the disk, be generous — a bigger spool buys a longer survivable outage:
```
metrana.init(..., disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=64 << 30))  # 64 GiB
```
Many small runs sharing one disk can collectively surprise you with how much space they hold. If your runs are small, restrict it so a forgotten spool can’t eat the volume:
```
metrana.init(..., disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=1 << 30))   # 1 GiB
```

There is no special “unlimited” sentinel — max_size_bytes is a byte count. To effectively uncap it, set it to whatever your disk can hold (a very large value); the spool then only stops when the filesystem does.

Configuration

DiskConfig flattens all spool tuning into one constructor. The defaults match the underlying engine:

Field	Default	Purpose
`dir` (positional)	(required)	Base spool directory.
`max_size_bytes`	`8 << 30` (8 GiB)	Spool size cap; reclaimed as segments drain.
`segment_size_bytes`	`100 << 20` (100 MiB)	Size of each on-disk segment file.
`compression`	`DiskCompression.Zstd`	On-disk compression (`Zstd` or none).
`zstd_level`	`3`	zstd compression level.
`io_max_retries`	`60`	Disk I/O retries before giving up; `None` retries forever.

metrana.init(..., disk=metrana.DiskConfig(
    "/var/spool/metrana",
    max_size_bytes=32 << 30,
    compression=metrana.DiskCompression.Zstd,
))

Passing disk="/path" (a bare string) uses all of these defaults. See the configuration reference.