RL metrics

Reinforcement-learning metrics use a two-level step: a major rl_step (the training/update step, which must not decrease) plus a minor step. Two helpers cover the common scales; the scale is implied by the function, so you never pass it explicitly. Both also accept labels and evaluation, which behave exactly as for standard metrics, and dtype, which selects the storage precision (float32 by default) exactly as for metrana.log.

Helper	Scale	Minor step
log_rl_episode	EPISODE	the episode
log_rl_environment_step	ENVIRONMENT_STEP	the within-episode env-step (auto-assigned or explicit)

Per-episode metrics

metrana.log_rl_episode(metric_name, value, rl_step, episode=None, env_id=None, labels=None, evaluation=False,
                       dtype=None)

The episode is the minor step. Omit it to auto-increment from the last logged episode:

metrana.log_rl_episode("episode_return", ep_return, rl_step=rl_step, episode=episode)

# auto-increment the episode
metrana.log_rl_episode("episode_return", ep_return, rl_step=rl_step)

Per-environment-step metrics

metrana.log_rl_environment_step(metric_name, value, rl_step, env_id=None, episode=None,
                                env_step=None, auto_start_from_one=False,
                                labels=None, evaluation=False, dtype=None)

The env-step axis counts across rl_steps, bounded by episode: each episode’s steps keep increasing from its first point until the episode ends, even when the episode spans many rl_steps.

By default (env_step=None) the axis is assigned automatically — steps are numbered server-side, so they survive process restarts with no client bookkeeping. Auto-assigned episodic steps start at 0; pass auto_start_from_one=True to number them from 1 instead. Alternatively, pass env_step to number the points yourself (see Explicit env steps).

Episodes are optional

You do not have to use episodes. The episode argument is optional, and many teams don’t structure their data into episodes at all:

# No episodes — one continuous env-step series per environment.
metrana.log_rl_environment_step("reward", reward, rl_step=rl_step)

With episode omitted there is a single env-step axis per environment that simply counts up forever. Pass episode only if you want per-episode segmentation; when you do, give the episode each point belongs to and every episode gets its own env-step axis.

One value, one env, or many envs

env_id defaults to a single environment (env_id=None ≡ one shared env). Pass a list to fan a batch across environments. The shape of value follows from how many envs and timesteps you’re logging:

Logging	env_id	value shape	meaning
one point, one env	"env0" or omitted	scalar	a single reward
many steps, one env	"env0" or omitted	1-D [T]	T env-steps for one env
one step each, many envs	["env0", …] (len M)	1-D [M]	one point per env
many steps, many envs	["env0", …] (len M)	2-D [M, T]	T env-steps for each of M envs

# single value, single env
metrana.log_rl_environment_step("reward", 0.7, rl_step=rl_step)

# bulk for one env: 4 env-steps
metrana.log_rl_environment_step("reward", [0.1, 0.3, 0.2, 0.5], rl_step=rl_step, env_id="env0")

# one reward each for 3 envs
metrana.log_rl_environment_step("reward", [0.7, 0.4, 0.9], rl_step=rl_step,
                                env_id=["env0", "env1", "env2"])

# 3 envs × 4 env-steps in one call
grid = [[0.1, 0.3, 0.2, 0.5],     # env0
        [0.2, 0.1, 0.0, 0.4],     # env1
        [0.5, 0.6, 0.4, 0.7]]     # env2
metrana.log_rl_environment_step("reward", grid, rl_step=rl_step, env_id=["env0", "env1", "env2"])

A scalar / 1-D value is single-env only

When env_id is a single id (or omitted), a scalar or 1-D value belongs to that one environment — values are not broadcast across environments. To log many envs, pass a list of env_ids and a [M] or [M, T] value. episode (and env_step / timestamp), in contrast, do broadcast: pass a scalar, 1-D, or 2-D matching the value.

Combining shapes with episodes

episode addresses the same [M, T] grid as value, in whichever of the three ranks fits your data:

episode shape	meaning
2-D [M, T]	the episode of every point — the general form; episodes may change mid-row
1-D [M]	one episode per env, constant across the row
scalar	every env is at this episode — lockstep setups only

Environments usually share rl_steps but not episode counters (each env finishes episodes at its own pace), so the per-env [M] and full [M, T] forms are the typical ones. The scalar broadcast is only correct when all environments genuinely advance episodes together, e.g. fixed-horizon vectorized envs that reset in lockstep.

# 3 envs × 4 env-steps; env1 finishes episode 4 mid-row and starts episode 5.
episodes = [[7, 7, 7, 7],
            [4, 4, 5, 5],
            [9, 9, 9, 9]]
metrana.log_rl_environment_step("reward", grid, rl_step=rl_step,
                                env_id=["env0", "env1", "env2"], episode=episodes)

The vectorized [M, T] form is the fast path — prefer one call over a per-env loop. See Logging at scale.

Explicit env steps

Pass env_step to number the env-step axis yourself — e.g. when your environment already tracks a step counter, or you want 1-based numbering with full control:

# 4 env-steps of episode 7, numbered by the caller.
metrana.log_rl_environment_step("reward", [0.1, 0.3, 0.2, 0.5], rl_step=rl_step,
                                episode=7, env_step=[12, 13, 14, 15])

env_step accepts the same shapes as episode, with one twist: a full [M, T] grid (or a 1-D row for a single env) gives the exact step of every point, while a scalar or per-env [M] array is the starting step of each (env, episode) segment — the rest of the segment continues contiguously from it, and an episode recurring later in the row continues its own run.

Explicit env steps must increase within their episode

Within one episode (or, without episodes, within the environment’s single axis), each explicit env step must be greater than the last one logged. A step at or below the episode’s last env step is deduplicated server-side — the call itself succeeds and the value is delivered, then dropped by the backend with only a client-side warning. The one sanctioned exception is dropping back to 0/1, which is the episode restart signal and erases the episode’s earlier steps.

Pick auto or explicit once per series

How the env-step axis is fed is pinned per (environment, metric) by the first call: mixing auto-assigned and explicit env_step calls on one series raises a validation error. Switching mid-run would corrupt the axis.

Episode restarts

Re-collecting an episode from its beginning (typically after a process restart) restarts it: the server discards the episode’s earlier env steps and the series starts over from the replay. The restart signal depends on how the axis is fed:

• Explicit env_step: the episode logs env-step 0 or 1 again — at any rl_step.
• Auto-assigned: the episode’s opening batch is replayed — it lands back on the episode’s first rl_step. (A from-scratch restart naturally has this shape; an episode appearing at a later rl_step continues instead.)

Episodes are restarted, never continued: a restarted process should re-collect episodes from their start (or move on to fresh episode numbers), not append to a half-logged one. Non-episodic series are the opposite — their axis never restarts and simply keeps counting across restarts.

A typical RL loop

import metrana

metrana.init(workspace_name="ws", project_name="proj", run_name="rl-001")
env_ids = [f"env{i}" for i in range(num_envs)]

try:
    for rl_step in range(num_updates):
        rollouts = collect_rollouts()   # rewards: [num_envs, T], episodes: [num_envs, T]

        metrana.log_rl_environment_step(
            "reward", rollouts.rewards, rl_step=rl_step,
            env_id=env_ids, episode=rollouts.episodes,
        )
        for env_i, ep_return in finished_episode_returns(rollouts):
            metrana.log_rl_episode("episode_return", ep_return,
                                   rl_step=rl_step, env_id=env_ids[env_i],
                                   episode=rollouts.last_episode[env_i])
finally:
    metrana.close()

RL data appears in the run’s Environments views. For the underlying model, see Reinforcement-learning concepts.

Resuming RL series

metrana.get_env_last_rl_step_and_episode(env_id) returns (last_rl_step, last_episode) for an environment (either may be None). It is seeded from the server at init(), so it’s the RL analogue of get_last_step for computing explicit steps after a resume:

last_rl_step, last_episode = metrana.get_env_last_rl_step_and_episode("env0")
next_rl_step = 0 if last_rl_step is None else last_rl_step + 1

Accurate only for an exclusive (single-writer) environment

get_env_last_rl_step_and_episode reports the furthest (rl_step, episode) the backend has seen for the environment. It’s exact when a single logger owns the environment. If multiple loggers write parallel episodes to the same environment (the parallel-episodes pattern), the returned value reflects whatever has been ingested so far and may not capture every writer’s progress — so don’t rely on it to derive non-overlapping episode numbers across those writers. It’s most useful when the resuming process had exclusive access to the environment.

Per-environment attributes

RL runs can also carry per-environment attributes (a distinct, env-scoped kind of metadata) via log_env_attributes. See Config & attributes.

Multiple writers

By default the RL helpers are ordered per env_id series, so each environment must be logged by exactly one process — unless you partition by episode. This matters when you shard environments across distributed workers; see Distributed logging.