Guarantees & retries

Metrana favours delivering your data over shaving microseconds off the hot path — while never blocking it forever. The defaults are chosen so a healthy run pays almost nothing and an unhealthy one fails loudly instead of silently dropping metrics.

Three knobs govern what happens as delivery comes under pressure: backpressure (queue full), retries (a send fails), and the delivery policy (everything has given up).

Backpressure: when the queue is full

backpressure_strategy (default "block") decides what an enqueue does when the in-process queue is full:

Strategy	Behaviour
"block" (default)	Wait for room rather than dropping. Bounded — the queue only stays full while delivery is stuck, which the retry policy itself bounds (and a disk spool drains quickly).
"drop_new"	Drop the new points. Protects the loop, but can lose data under sustained pressure.
"raise"	Raise MetranaEventQueueFullError.
"spool_then_raise"	Requires a disk spool: a timed-out wait triggers the spool failover by itself and waits disk_spool_retry_timeout_secs (default 5) longer for it to unblock the queue; raises only if even local disk could not absorb the data.

The non-blocking policies wait up to enqueue_timeout_secs (default 1) for the queue to drain before they drop, raise, or spool.

"spool_then_raise" = bounded latency without loss

With a spool configured, the queue-full trade-off disappears: a timed-out enqueue (or flush) fails over to local disk within moments, logging resumes at disk speed, and the recovery channel delivers everything once the backend catches up. Logging never blocks past enqueue_timeout_secs + disk_spool_retry_timeout_secs and nothing is lost. For precise manual control (e.g. failing over ahead of a known network cutover), "raise" paired with metrana.failover_to_spool() in the exception handler does the same switch by hand.

Retries: when a send fails

max_send_retries (default 15) controls how many times a failed gRPC send is retried, with exponential backoff from send_retry_initial_backoff_secs (default 0.05) to send_retry_max_backoff_secs (default 15.0) — about 1.7 minutes total at the defaults.

When the retry budget is spent, the logger fails over to the disk spool if one is configured; otherwise the delivery policy below applies. max_send_retries=None retries gRPC forever.

Delivery policy: when everything gives up

delivery_policy (default "all_or_crash") decides what happens once every channel (gRPC and, if configured, the disk spool) has given up:

Policy	Behaviour
"all_or_crash" (default)	Stop the logger so the next log / flush / close raises MetranaSenderError. For an experiment tracker, a loud failure beats a silent hole.
"tolerate_loss"	Drop the data and keep the process running — for callers who would rather lose metrics than the job.

What the defaults actually guarantee

With block + all_or_crash and the spool disabled (disk=None, or the default spool unavailable):

• Transient outages survive — they’re retried.
• The only silent-loss path is a hard process kill (kill -9, or exiting without close()).
• A sustained outage (longer than the ~2-min retry budget) still loses the buffered window — but loudly: the loop blocks, then all_or_crash stops the logger and the next call raises.

So without a spool, “favouring delivery” means fail loud, not survive.

The disk spool closes the gap — and is on by default

A disk spool turns a sustained outage into a transparent failover — data is written to disk and delivered when the backend returns. Every run gets one under ~/.metrana/spools unless you pass disk=None; if that default location is unusable, init() warns and the run is back to fail-loud (an offline start raises instead, since it has nowhere else to log). (Or opt into tolerate_loss / drop_new to drop-and-continue instead of block-then-crash.)

Choosing a resilience profile

The knobs compose into two recommended spool-backed profiles. Pick by what your job can afford — waiting (delivery favours the wire, the loop may slow down) or spooling eagerly (every wait is bounded, more data takes the disk → replay path). Both size the spool explicitly — see Capacity for matching max_size_bytes to your disk.

Durability-first: block and wait

For jobs with no clock against them — no spot preemption, no tight step budget — that would rather slow down than divert data while the backend is down or applying backpressure:

metrana.init(
    ...,
    backpressure_strategy="block",         # the default: wait for the queue, never drop
    disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=32 << 30),
)
# ...
metrana.flush(timeout_secs=None)      # or the generous 900 s default
metrana.close(close_timeout=None)     # unbounded: let a paced drain finish

Delivery favours the wire: an outage rides the full retry budget (~2 min) before failing over to the spool, server-side backpressure just paces the loop, and unbounded (or default 900 s) timeouts let a slow drain finish. Nothing raises mid-run; the spool exists for genuine outages.

Bounded-latency: spool eagerly

For jobs on a clock — spot/preemptible nodes, tight step budgets — the spool absorbs pressure instead of your loop. Every knob that can wait gets a short bound, and timed-out waits land on the spool:

metrana.init(
    ...,
    backpressure_strategy="spool_then_raise",    # timed-out waits fail over by themselves
    enqueue_timeout_secs=1.0,           # tolerance for a slow primary (default)
    disk_spool_retry_timeout_secs=5.0,  # window for the spool takeover to unblock (default)
    ack_stall_timeout_secs=10.0,        # detect a stalled-but-alive backend fast (default 60)
    max_send_retries=3,                 # fail over after seconds, not minutes (default 15)
    close_spool_reserve_secs=5.0,       # tail of a bounded close reserved for the spool (default)
    max_pending_requests=3,             # small in-flight window -> fast drains (default 30)
    disk=metrana.DiskConfig("/var/spool/metrana", max_size_bytes=8 << 30),
)
# ...
metrana.flush(timeout_secs=10)          # a timeout fails over and finishes on the spool
metrana.close(close_timeout=60)         # short close: the reserve spools the tail

Enqueues and flushes are bounded by enqueue_timeout_secs/the flush timeout plus disk_spool_retry_timeout_secs; a stalled backend is cut loose after roughly ack_stall_timeout_secs × max_send_retries; and a bounded close ends with the tail durable on disk thanks to the emergency reserve. The trade: the spool engages on any sustained pressure, so more data arrives via recovery/replay instead of live. See the preemption guidance for sizing the close window to a grace period.

Below these: the default, and the lossy corners

• The default (block + all_or_crash, no spool) is the “fail loud” contract described above — transient outages survive, sustained ones stop the logger with an error rather than a silent hole.
• "drop_new" / tolerate_loss without a spool keep the loop running at the cost of data — for callers who would rather lose metrics than the job. With a spool configured they rarely make sense: the spool already keeps the loop fast without dropping anything.

Rendering errors are separate

rendering_error_strategy (default "warn") governs only the Python rendering pipeline’s frame-encoding errors ("silent", "warn", "raise_on_log", "raise_on_close"). Engine delivery of metrics and attributes is governed by delivery_policy, not this. See Renderings.

See also

• Lifecycle: flush, close, shutdown — durability barriers and timeouts.
• Disk spool — the recommended durability layer.
• Errors — the exception types these policies raise.