Resuming & forking runs

Resuming a run

When a process re-initializes a run that already exists, Metrana can resume it — continuing the same run rather than creating a new one. This is controlled by resume_strategy on init():

resume_strategy	behaviour
"never" (default)	Only a process of the same job resumes (matched by orchestration_id); a genuinely different job hitting the same run name errors instead of corrupting it.
"allow"	Resume the existing run if it exists, otherwise create a new one.

Under "never", a different launch hitting the same run name raises MetranaRunAlreadyExistsError (or the more specific MetranaOrchestrationIdMismatchError) — see Errors. This is what makes the default safe for distributed jobs, where many sibling processes share one run; see Distributed logging for how orchestration_id ties the siblings together.

Continuing the step axis after a resume

On resume, init() seeds the last step from the server, so you can pick up where the previous process left off:

metrana.init(workspace_name="ws", project_name="proj", run_name="run-001", resume_strategy="allow")

last = metrana.get_last_step("loss")
next_step = 0 if last is None else last + 1
for step in range(next_step, next_step + more_steps):
    metrana.log("loss", loss, step=step)

The RL equivalent is get_env_last_rl_step_and_episode. See also Retrieving the last step.

Graceful stop for resume

To stop a run cleanly so it can be resumed later — e.g. on a spot-instance preemption signal — call metrana.shutdown_logger() instead of close(). It flushes in-flight data and stops the engine but leaves the run open, so the next process can init() and resume it.

Forking a run

Forking creates a new run that branches off an existing one at a chosen step — the new run inherits the parent’s history up to that point, then diverges. Pass both fork_from_run and fork_at_step to init():

metrana.init(
    workspace_name="ws",
    project_name="proj",
    run_name="run-001-fork-from-2000",   # the new (child) run
    fork_from_run="run-001",             # the parent run to branch from
    fork_at_step=2000,                   # inclusive step to fork at (major/RL step for RL)
)

• fork_from_run — the name of the existing run to branch from. Requires fork_at_step.
• fork_at_step — the inclusive step to fork at (the major step for RL runs). Requires fork_from_run.

The child run starts logging from after fork_at_step, while still showing the parent’s data up to that point — useful for hyperparameter branches, warm-restarts, or exploring “what if I’d changed X at step N”.

What is inherited on a fork

What the child inherits from the parent depends on the kind of series.

Standard and per-episode series

Plain float series (log / log_distributed) and per-episode RL series (log_rl_episode) are inherited up to and including fork_at_step. The child continues them from after the fork step.

Episodic RL series (log_rl_environment_step with an episode)

Episode-segmented env-step series fork at episode granularity, relative to the fork RL step:

• Episodes that ended at or before the fork RL step are inherited whole.
• Episodes that started at or before the fork RL step but continued past it are inherited as the partial slice up to the fork — the portion after the fork is not carried into the child.
• For each environment, the child should continue from the next episode to avoid overshadowing the episodes inherited from the parent — unless overshadowing them is exactly what you intend.

Continue from the next episode per environment

After a fork, query each environment’s resume point (e.g. get_env_last_rl_step_and_episode) and start the child’s logging at the following episode, so inherited parent episodes aren’t overwritten.

Attributes

• Run attributes (config, tags, description) are forked — the child inherits the parent’s run-level metadata.
• Environment attributes are not forked (as of now). The expectation is that you initialize each environment afresh — log the full environment configuration for the next episode the child will run.