Distributed logging

When several processes (e.g. distributed-training ranks, or parallel rollout workers) log into one run, three things matter: they must agree on the run, they must use the right log function for shared series, and RL series need an exclusive owner per (environment, episode).

1. Agree on the run (orchestration_id)

Every process that should share a run needs the same orchestration_id. If you don’t pass one, it is resolved automatically, in order, from:

1. METRANA_ORCHESTRATION_ID
2. the framework job ids TORCHELASTIC_RUN_ID / SLURM_JOB_ID / RAY_JOB_ID
3. a random token (which only descendants that inherit the environment will match)

The resolved value is published back to METRANA_ORCHESTRATION_ID, so forked/spawned children inherit it. With resume_strategy="never" (the default), the first process creates the run and the rest resume it by matching this identifier; a genuinely different job hitting the same run name errors instead of corrupting it (see Resuming a run).

# torchrun / Slurm / Ray: nothing to do — the framework job id is picked up automatically.
metrana.init(api_key="...", workspace_name="ws", project_name="proj", run_name="run")

# Custom launcher: pass a token shared by all workers of the job.
metrana.init(..., orchestration_id="job-2025-06-23-abc")

2. Choose the right log function for shared series

Use metrana.log_distributed(...) (instead of metrana.log(...)) when multiple processes write to the same series — for example all ranks logging a global loss. It uses unordered, merge semantics so concurrent writers don’t conflict. Provide an explicit step (the global training step) so points from different ranks align on the same axis:

metrana.log_distributed("loss", loss, step=global_step)

Use plain metrana.log(...) for series owned by a single writer — it is ordered and can auto-increment. Pin logger_id (e.g. one per rank) if you want the backend to distinguish a restarted writer from a genuinely new concurrent one.

Series ownership	Function	Semantics
One writer	log	Ordered; can auto-increment the step.
Many writers	log_distributed	Unordered/merge; pass an explicit global step.

log_distributed series support a limited set of aggregations

Because a distributed (unordered) float series merges points from many writers, the backend computes only min / max / first / last aggregations over it — not mean/sum/std and the like. If you need the full set of aggregating statistics, that’s available for parallel environments / episodes (the RL merge path below), which do support all of them. Plan which writer pattern you use accordingly.

3. RL metrics need an exclusive owner per (environment, episode)

The RL functions (log_rl_episode, log_rl_environment_step) are ordered, and the unit of ownership is the (environment, episode) pair: each episode of each environment must be logged by exactly one process. Sharding whole environments across ranks (e.g. a vectorized env split over workers) satisfies this automatically. Sharding episodes of one environment across parallel rollout workers — each worker running its own episodes of env_id="cartpole" — works too, provided each worker passes a unique episode_id.

• log_rl_episode — pass an explicit episode. This switches the series to unordered (merge) semantics: you own the episode numbering, and parallel workers’ points merge cleanly instead of contending for one auto-increment cursor.
• log_rl_environment_step — nothing special: log normally with episode=... set. Env steps are numbered within each episode, so writers that own different episodes never contend.

# each rollout worker logs only the episodes it ran; env0 is shared by many workers
metrana.log_rl_episode("episode_return", ep_return, rl_step=rl_step,
                       env_id="env0", episode=my_episode)        # explicit -> merge

metrana.log_rl_environment_step("reward", rewards, rl_step=rl_step,
                                env_id="env0", episode=my_episode)

Episode restarts are the main risk with parallel loggers

An episode that logs its first env-step again is treated as an episode restart: the server discards the episode’s earlier points and starts it over (this is how a crashed-and-restarted worker recovers). Two processes writing the same episode of one environment therefore don’t just interleave — the second writer looks like a restart and silently wipes what the first one logged. Keep episode ownership disjoint.

Plain log / log_distributed float series have no per-env restriction (log_distributed is explicitly built for many writers on one series).

Create-only attributes and the disk spool

Two related rules for the multi-process case:

• Create-only attributes. config / tags / description / git SHA from init() are applied only by the creating process, so resuming siblings never clobber them. See Config & attributes.
• Disk spool isolation. Concurrent writers must never share one spool directory — give each its own container/volume or a distinct disk= path. See Disk spool → Multiple processes.