Logging at scale

When you log a lot — many series, high step rates, many environments — a few habits keep the overhead negligible.

Log in bulk, not point-by-point

Every log call crosses from Python into the engine. One call with an array amortises that crossing far better than a loop of scalar calls.

# slow: N crossings
for i, v in enumerate(values):
    metrana.log("loss", v, step=i)

# fast: one crossing, N points
metrana.log("loss", values, step=0)        # steps 0, 1, 2, ...

For RL, prefer the vectorized form over a per-env loop — it takes a [num_envs, T] block in a single call:

# fast: 8 envs x 128 steps in one call
metrana.log_rl_environment_step("reward", rewards_8x128, rl_step=rl_step,
                                env_id=env_ids, episode=episodes_8x128)

See Standard metrics and RL metrics. Contiguous arrays already in the target storage dtype (float32 by default) are handed to the engine without a copy, so wide batches are cheap.

Don’t flush on the hot path

flush() is a synchronous barrier — it waits for delivery/durability, so calling it every step serialises your loop against the network. Flush at checkpoints (or every few minutes), not every step; let the background engine batch and stream everything in between.

Give the engine room

Under sustained high rates, raise the engine’s buffers:

• queue_capacity — more buffer before backpressure.
• max_pending_requests — more in-flight streams.
• batch_max_age_secs — how long points coalesce before a send.

Then watch get_metrics() — compare added vs sent and check *_dropped — to confirm you’re keeping up. See Tuning & observability for the knobs and counters.