Offline logging

An offline run never talks to the ingestion service while it runs. Everything you log lands on the disk spool, and the run is created on the backend later, when you replay the spool from a machine that can reach it. It is how you use Metrana from an air-gapped GPU cluster.

Choosing the start mode

init_mode (also METRANA_INIT_MODE) picks how init() brings the run up:

init_mode	Behaviour
"online_only" (default)	The handshake must succeed within init_timeout_secs; any failure raises MetranaLoggerInitError. The one exception is a restart: if earlier processes of the run left spools on this node, a failed handshake is treated as an outage and the process continues on the spool (an offline leftover keeps it offline).
"offline"	Never contact the backend. Requires a disk spool.

# air-gapped cluster: no backend, spool everything
metrana.init(api_key="...", workspace_name="ws", project_name="proj", run_name="run-042",
             init_mode="offline", disk="/scratch/metrana-spool")

metrana.log("loss", 0.42)              # lands on the spool
print(metrana.is_offline())            # True
metrana.close()                        # logs where the spool is, no warning

There is deliberately no automatic fallback from online to offline: a mistyped endpoint or a revoked key fails init() loudly instead of silently producing an offline run whose data only arrives after a replay. Offline is an explicit choice for a cluster that cannot reach the backend. If the service is merely slow to come up when your job starts, raise init_timeout_secs; a backend outage during the run is what the disk spool is for.

What is different offline

• No run id yet. metrana.is_offline() is True; the run does not exist on the backend until replay.
• Rendering upload is disabled, since videos would have no run to link to: finalized renderings stay under the rendering output directory and are not deleted.
• Forks are refused. A forked run starts from its parent’s cursors, which only the backend knows, so init_mode="offline" with fork_from_run raises. (A process os.fork() is fine: the child gets its own writer spool under the run directory.)
• The spool is required. With the default spool unavailable (the directory is unusable) init() raises MetranaSpoolUnavailableError rather than continuing without one.
• Its cap is the whole run’s budget. Space is normally reclaimed by the recovery channel as it delivers, and offline that never runs — so nothing is freed until you replay, and max_size_bytes (8 GiB by default) bounds everything the run will ever log, not a rolling window. Reaching it stops the run under the delivery policy. Starting offline on the default cap logs a warning for this reason. See Capacity for the arithmetic; the short version is that 8 GiB is years of ordinary ML logging and about a day of high-fanout RL.
• A run’s mode is fixed for its lifetime. A process that restarts on the same spool continues the offline run — step cursors included — whatever init_mode it configures. The reverse is refused: if the spool holds a run that was already created on the backend, init_mode="offline" raises rather than starting a second run alongside it.
• Everything else — steps, RL series, attributes, tags, histograms, scatter plots, flush, shutdown_logger — works exactly as online. close() logs how many spool segments await delivery, at info level.

Delivering an offline run

Copy the spool to a machine that can reach the backend (or wait until the cluster can), then replay it:

import metrana

info = metrana.replay_spool(
    "/scratch/metrana-spool",          # the BASE disk path; omit for the default spool
    workspace_name="ws",
    project_name="proj",
    run_name="run-042",
    api_key="...",
)
print(info.metrics.float_points_sent, "points delivered")

The replay performs the handshake the run skipped, with the parameters the run was started with (they are persisted next to the spool): it creates the run, then delivers every spooled record under the run id the backend allocated, and closes the run if the offline process closed it. A validation failure from the backend — an invalid name, a rejected key — surfaces from replay_spool as the matching init error, and the spool is left intact for another attempt. A fully delivered spool is consumed: its files are removed, so replay is the last step for a finished run. Running it again has nothing to replay, and a later init() on the same disk path starts a fresh, online logger rather than resuming the run that was already delivered.

Distributed offline jobs

Several processes can start the same run offline — on one node they share the run’s spool directory with a writer directory each, on several nodes each has its own — and the usual distributed logging rules apply, plus one that matters more than usual: they must share an orchestration_id. One replay_spool per run directory delivers every writer in turn: the first creates the run, and every later one attaches to it because the run carries the job’s orchestration id. A run of the same name owned by a different job (or by nobody — no orchestration id) is refused, never overwritten. Since the framework job ids that normally resolve the orchestration id may not exist on the replaying machine, the id persisted with each spool is what the replay uses; you need not pass it again.