RL environments

In a reinforcement-learning run, the environments are a level below the run — each with its own episode and step counters, its own config attributes, and its own series. These methods read that level. For the concepts behind it, see Environments (RL).

Where a run is

info = client.get_rl_info("rl-001")
info.environments_count, info.latest_episode, info.latest_rl_step

One cheap call, and usually the first one: it tells you the run’s extent before you ask for anything sized by it.

It returns an RLInfo:

Field	Meaning
environments_count	How many environments the run logged.
latest_episode	Highest episode across all environments.
latest_rl_step	Highest RL step across all environments.
is_rl	Property: False for a run that logged no RL environment at all.

The two latest_* fields are maxima across environments, so an individual environment may sit behind them — don’t assume every environment reached latest_episode.

Use is_rl to branch before asking RL questions of a run that may be a plain ML run:

info = client.get_rl_info(run)
if not info.is_rl:
    return          # nothing environment-shaped to fetch

The environments table

envs = client.fetch_rl_envs("rl-001")

One row per environment: environment_id, latest_episode, latest_rl_step, current_episode, then attribute columns, then float64 summary columns. It takes the same filter / order / column arguments as fetch_runs — including the label rules, so a selector whose labels match several of an environment’s series gives one column aggregated across them:

from metrana.query import AttrFilter, SeriesOrderBy, ENV_ID

client.fetch_rl_envs(
    "rl-001",
    filter=AttrFilter("cfg/policy") == "ppo",
    order_by=SeriesOrderBy("reward", "avg").with_scale("ENVIRONMENT_STEP").desc(),
    attributes=["cfg/policy", "cfg/seed"],
    max_environments=50,
)

Note the explicit with_scale. RL series are written at ENVIRONMENT_STEP or EPISODE, never at the ML_STEP a scale-less selector means, so the client rejects a scale-less series selector here rather than returning an empty table — see RL series.

ENV_ID, ENV_LATEST_EPISODE and ENV_LATEST_RL_STEP name the reserved fields, so you can filter and sort on them like any attribute:

client.fetch_rl_envs("rl-001", filter=AttrFilter(ENV_ID).matches("^worker-"))

The two timeline axes

The environments table exists on two axes, and at= picks the point on one of them:

from metrana.query import Episode, RLStep

client.fetch_rl_envs("rl-001", at=Episode(12))    # as of episode 12
client.fetch_rl_envs("rl-001", at=RLStep(400))    # as of RL step 400

An episode summary aggregates over the environment steps of that episode; an RL step summary aggregates over the rollout indexes of that step. The two axes interleave rather than nest: an episode can span more than one RL step, and an RL step can contain more than one episode.

One consequence worth knowing: on the RL-step axis this table shows each environment’s latest episode’s value at that step — not an aggregate over every episode the step touched. (The series fetches differ: anchored at an RL step they return all the episodes’ parts inside it.)

Omitting at means the run’s latest episode, which is what you want for “where is this run now”. The anchor costs one extra call when omitted, since the latest episode has to be looked up first — pass it explicitly in a loop.

Environment config attributes

Environments carry their own attributes, logged with log_env_attributes and optionally keyed by episode:

client.list_rl_env_attributes("rl-001", "envA")
# {'cfg/policy': 'ppo', 'cfg/seed': 7}

client.list_rl_env_attributes("rl-001", "envA", episode=3)   # as they were at episode 3

To discover which paths exist before filtering on them, use list_rl_env_attribute_definitions.

Series inside the environments

Reading the environments’ metrics is its own page — see RL series.