Lifecycle: flush, close, shutdown

Three calls manage durability and shutdown. All three accept a timeout bounding how long they wait (timeout_secs for flush; close_timeout for close / shutdown_logger); pass None to wait without bound. The defaults are finite so an unreachable backend can never hang interpreter shutdown forever.

Call	What it does	Run state after
flush(timeout_secs=180)	Durability barrier — returns only once everything logged before it is durable.	Run stays open, logger usable.
close(close_timeout=180)	Flush, mark the run closed, stop the engine.	Run closed.
shutdown_logger(close_timeout=180)	Flush, stop the engine, but leave the run open.	Run open (resumable).

Why the default is 180 seconds

The 180 s default isn’t arbitrary. If the backend happens to go down exactly as you call flush / close / shutdown_logger, the data must first exhaust the gRPC retry budget (~2 min) before it can fail over to the disk spool — so the default leaves headroom for that worst-case failover to land the data safely on disk instead of being cut short. Shorter timeouts trade that safety for a faster exit; see Sizing timeouts for strong guarantees.

flush — a durability barrier

metrana.flush(timeout_secs=180)

flush returns only once everything logged before it is durable — acknowledged by the backend, or (with a disk spool) fsync’d to it. It rides out a transient outage (a reconnect does not fail it) and raises only if that data is actually dropped (MetranaSenderError) or the timeout elapses (MetranaTimeoutError — not loss; the data is still in flight or on the spool). The logger stays usable afterwards.

Call flush at checkpoints. A flush() next to your checkpoint write makes the metrics leading up to the checkpoint durable together, so a resumed run lines up with its metrics:

if step % checkpoint_every == 0:
    save_checkpoint(model)
    metrana.flush()          # metrics up to this checkpoint are now durable

Don’t flush every step

flush is a synchronous barrier — it serialises your loop against the network. Flush at checkpoints or every few minutes, not every step. See Logging at scale.

close — close the run

info = metrana.close(close_timeout=180)

close flushes queued points, marks the run closed, and shuts the background engine down, returning a CloseInfo. Always call it — the engine runs on a daemon thread, so if the interpreter exits without close(), queued-but-unsent points are lost (a warning is emitted at exit). Use it directly or via try/finally.

In distributed runs, the run closes when the last logger closes

close() from one process flushes and stops that logger, but the run is only marked closed once the last participating logger closes. So in a multi-process job every process can safely close() when it’s done; the run stays open until the final one exits. See Distributed logging.

close_timeout=0 means accept data loss

close() drains both metrics and rendering frames. A very short close_timeout (e.g. 0) abandons anything not yet delivered — that’s metric data loss, not just dropped frames — so reach for it only when you genuinely want to bail out fast. If your goal is merely to not wait on rendering frames at close, use init(skip_drain_render_on_close=True) instead: it drops only queued frames while metric delivery still gets its full window. See Renderings → closing.

shutdown_logger — stop without closing

info = metrana.shutdown_logger(close_timeout=180)

Like close, but leaves the run open so a later process can resume it. Use it for a graceful, resumable stop — e.g. on a spot-instance preemption signal, call it to flush in-flight data and stop cleanly instead of being killed mid-send, then init() the same run when the job restarts. See Resuming a run.

CloseInfo

close(), shutdown_logger(), and replay_spool() return a CloseInfo:

info = metrana.close()
print(info.metrics)                  # final self-metrics snapshot
if info.has_undelivered_spool:       # data still on the disk spool, not yet seen by the backend
    print(f"{info.leftover_segments} spool segments ({info.leftover_bytes} bytes) left to replay")

• metrics — final self-metrics snapshot (see Tuning & observability).
• has_undelivered_spool — True when the disk spool still holds data the recovery channel could not deliver within the close window; a future resume or replay_spool will deliver it. Always False when no disk spool is configured.
• leftover_segments / leftover_bytes — how much remains on the spool.

Sizing timeouts for strong guarantees

During a backend outage, data is not spooled instantly — it first exhausts the gRPC retry budget (up to ~2 minutes at the defaults), then fails over to the disk spool. So a flush / close / shutdown_logger timeout shorter than that window can elapse before the data ever reaches disk.

The two surface differently:

• flush raises MetranaTimeoutError — benign; the data is still in flight, just re-flush or close.
• close / shutdown_logger raise MetranaClosingError only if their deadline elapses with data still not durable (neither acknowledged nor written to the spool) — which means likely loss. If the data reached the spool before the deadline they do not raise: they return a CloseInfo and log that the data is safe on disk, awaiting delivery.

For strong guarantees, configure a disk spool and give these calls a timeout at least as long as the failover window (≥ ~2–3 minutes). An outage at close-time then lands the data on disk — recoverable on the next resume or via replay_spool — instead of cutting the failover short.