Lifecycle: flush, close, shutdown

Three calls manage durability and shutdown. All three accept a timeout bounding how long they wait (timeout_secs for flush; close_timeout for close / shutdown_logger); pass None to wait without bound. The defaults are finite so an unreachable backend can never hang interpreter shutdown forever.

Call	What it does	Run state after
flush(timeout_secs=900)	Durability barrier — returns only once everything logged before it is durable.	Run stays open, logger usable.
close(close_timeout=900)	Flush, mark the run closed, stop the engine.	Run closed.
shutdown_logger(close_timeout=900)	Flush, stop the engine, but leave the run open.	Run open (resumable).

Why these defaults

The shared 900 s default is not arbitrary. All three calls drain the same queue under the same conditions, and two windows stack in the worst case. First the outage window: if the backend goes down exactly as you call, the data must exhaust the gRPC retry budget (~2 min) before it can fail over to the disk spool. Second, backpressure: when the ingestion backend is behind, the server deliberately paces senders (up to ~5 s of added latency per in-flight message) so its workers can catch up — draining a full send window under that pacing can healthily take several minutes. A deadline that expires mid-drain reports a timeout for data that was on its way (for flush that is benign — flush again and the wait resumes; for close it abandons the drain). Shorter timeouts trade that safety for a faster return; see Sizing timeouts for strong guarantees.

flush — a durability barrier

metrana.flush(timeout_secs=900)

flush returns only once everything logged before it is durable — acknowledged by the backend, or (with a disk spool) fsync’d to it. It rides out a transient outage (a reconnect does not fail it) and raises only if that data is actually dropped (MetranaSenderError) or the timeout elapses (MetranaTimeoutError — not loss; the data is still in flight or on the spool). The logger stays usable afterwards.

Call flush at checkpoints. A flush() next to your checkpoint write makes the metrics leading up to the checkpoint durable together, so a resumed run lines up with its metrics:

if step % checkpoint_every == 0:
    save_checkpoint(model)
    metrana.flush()          # metrics up to this checkpoint are now durable

Don’t flush every step

flush is a synchronous barrier — it serialises your loop against the network. Flush at checkpoints or every few minutes, not every step. See Logging at scale.

close — close the run

info = metrana.close(close_timeout=900)

close flushes queued points, marks the run closed, and shuts the background engine down, returning a CloseInfo. Always call it — the engine runs on a daemon thread, so if the interpreter exits without close(), queued-but-unsent points are lost (a warning is emitted at exit). Use it directly or via try/finally.

In distributed runs, the run closes when the last logger closes

close() from one process flushes and stops that logger, but the run is only marked closed once the last participating logger closes. So in a multi-process job every process can safely close() when it’s done; the run stays open until the final one exits. See Distributed logging.

close_timeout=0 means accept data loss

close() drains both metrics and rendering frames. A very short close_timeout (e.g. 0) abandons anything not yet delivered — that’s metric data loss, not just dropped frames — so reach for it only when you genuinely want to bail out fast. If your goal is merely to not wait on rendering frames at close, use init(skip_drain_render_on_close=True) instead: it drops only queued frames while metric delivery still gets its full window. See Renderings → closing.

shutdown_logger — stop without closing

info = metrana.shutdown_logger(close_timeout=900)

Like close, but leaves the run open so a later process can resume it. Use it for a graceful, resumable stop — e.g. on a spot-instance preemption signal, call it to flush in-flight data and stop cleanly instead of being killed mid-send, then init() the same run when the job restarts. See Resuming a run.

Preemption grace periods are shorter than the default

The 900 s default assumes you can afford to wait. Preemption notices are much shorter — ~2 minutes on AWS spot, as little as ~30 s elsewhere. For constrained environments:

• Use the bounded-latency profile (backpressure_strategy="spool_then_raise" + short timeouts + a sized spool) — it bounds every wait in the run, not just the close.
• Pass an explicit close_timeout sized to your provider’s grace period, and configure a disk spool — the run is resumable, so anything already spooled is delivered by the next resume.
• Lower init(max_pending_requests=...) (default 30) to e.g. 3: it caps how much data is in flight at any moment, so even a backpressure-paced drain completes in seconds rather than minutes, and an outage at close-time leaves the failover only a few messages to land on the spool — typically well under a minute end to end. The trade is peak logging throughput, which is why it is not the default; it fits jobs that log modest volumes from preemptible nodes.

Two mechanisms make short deadlines safe (with a spool configured):

• Emergency spool reserve (close_spool_reserve_secs, default 5): once a bounded close/shutdown window has only the reserve left, everything still undelivered is written straight to the spool — no gRPC retries, no recovery channel — and delivered by the next resume. The reserve is capped at half the close window, so delivery always gets the first half; even close_timeout=30 on a 30-second grace period ends with the data durable on disk.
• Ack-stall watchdog (ack_stall_timeout_secs, default 60): a backend that answers keepalives but stops acking is treated as failed — reconnect, then the usual retry/failover to the spool — so log calls can never block indefinitely on a stalled-but-alive connection. The bound is real but generous: each stalled stream burns one retry and only a real ack resets the budget, so the worst case to reach the spool is ack_stall_timeout_secs × max_send_retries — ~15 minutes at the defaults, plus backoffs. Latency-sensitive callers should lower one of those knobs, or skip the wait entirely with failover_to_spool() / backpressure_strategy="spool_then_raise".

failover_to_spool — bounded latency on demand

For callers that cannot tolerate long tail latencies anywhere — not just at close — the spool can be engaged manually. metrana.failover_to_spool() switches delivery to the disk spool immediately: everything buffered lands on disk, logging and flushes then complete at local-disk speed, and the recovery channel drains the spool back to the server in the background — the switch is temporary and self-healing, returning to gRPC once the backend catches up.

The bounded-latency pattern: a non-blocking backpressure strategy, short flush timeouts, and the failover in the timeout handler — logging never blocks past your budget, and nothing is lost:

from metrana.utils.exceptions import MetranaTimeoutError

metrana.init(..., backpressure_strategy="raise",
             disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=8 << 30))
try:
    metrana.flush(timeout_secs=2)
except MetranaTimeoutError:
    metrana.failover_to_spool()      # durable on local disk within moments
    metrana.flush(timeout_secs=5)

Importing exceptions

Exception types live in metrana.utils.exceptions, not on the metrana package root — metrana.MetranaTimeoutError raises AttributeError. See Errors.

It returns without waiting for the switch — follow with flush() as the durability barrier — and raises MetranaValidationError if no spool is configured.

Prefer the declarative form when the policy is uniform: init(backpressure_strategy="spool_then_raise", disk=...) makes every timed-out log_*/flush wait trigger this failover by itself and wait disk_spool_retry_timeout_secs (default 5) longer for the spool to unblock it — the same bounded latency without the try/except wiring. The manual call remains for precise control, e.g. failing over ahead of a known network cutover.

CloseInfo

close(), shutdown_logger(), and replay_spool() return a CloseInfo:

info = metrana.close()
print(info.metrics)                  # final self-metrics snapshot
if info.has_undelivered_spool:       # data still on the disk spool, not yet seen by the backend
    print(f"{info.leftover_segments} spool segments ({info.leftover_bytes} bytes) left to replay")

• metrics — final self-metrics snapshot (see Tuning & observability).
• has_undelivered_spool — True when the disk spool still holds data the recovery channel could not deliver within the close window; a future resume or replay_spool will deliver it. Always False when no disk spool is configured.
• leftover_segments / leftover_bytes — how much remains on the spool.

Sizing timeouts for strong guarantees

During a backend outage, data is not spooled instantly — it first exhausts the gRPC retry budget (up to ~2 minutes at the defaults), then fails over to the disk spool. So a flush / close / shutdown_logger timeout shorter than that window can elapse before the data ever reaches disk.

The two surface differently:

• flush raises MetranaTimeoutError — benign; the data is still in flight, just re-flush or close.
• close / shutdown_logger raise MetranaClosingError only if their deadline elapses with data still not durable (neither acknowledged nor written to the spool) — which means likely loss. If the data reached the spool before the deadline they do not raise: they return a CloseInfo and log that the data is safe on disk, awaiting delivery.

For strong guarantees, configure a disk spool and give these calls a timeout at least as long as the failover window (≥ ~2–3 minutes). An outage at close-time then lands the data on disk — recoverable on the next resume or via replay_spool — instead of cutting the failover short.