RL environments

Reinforcement-learning runs hold data per environment, indexed by RL step and episode (see Reinforcement learning concepts and, for how the data is written, Reinforcement learning data). These RPCs read it.

Run overview

GetRLInfo (rl-info:get) returns a run’s environment count and its latest episode and RL step across all environments. A count of 0 means the run has no RL data.

Listing environments

ListRLEnvironments (rl-environments:list) lists a run’s environments as a table: filter and sort them, and return environment attributes and series summaries for each.

{
  "workspace": "my-workspace",
  "project": "control",
  "run": "ppo-seed3",
  "scaleSelector": {"episode": "200"},
  "filter": "attr(config/map) == \"maze-large\" AND episode_return(scale=EPISODE):max > 150",
  "orderBy": "episode_return(scale=EPISODE):max DESC",
  "attributePaths": ["config/map", "config/difficulty"],
  "seriesSummaries": ["episode_return(scale=EPISODE):max", "episode_length(scale=EPISODE):avg"],
  "limit": "50"
}

scaleSelector pins the point in each environment’s history the listing describes — an episode or an rlStep — and every filter, attribute and summary is evaluated there. An environment that has not reached the requested episode uses its latest one; each result reports the episode it used as currentEpisode, alongside its overall latestEpisode and latestRlStep.

Filters and orderings accept environment attributes via attr(…), series selectors, and three special fields:

Field	Type
attr(environment_id)	string
attr(latest_episode)	integer
attr(latest_rl_step)	integer

Paging uses limit, offset and totalCount, as for the other listings.

Environment attributes

Environment attributes are versioned by episode: a value logged at one episode stays in effect until it is logged again. ListRLEnvAttributes and BrowseRLEnvAttributes read one environment’s attributes: without episode, the latest values; with it, the values in effect at that episode.

{"workspace": "my-workspace", "project": "control", "run": "ppo-seed3", "environment": "env-07", "episode": "120"}

Discovering environment data

RL series and attributes have their own discovery RPCs, shaped like the general ones:

Discover	Within one run	Across runs
Environment attribute definitions	ListRLAttributesDefinitions, BrowseRLAttributesDefinitions (optionally one environment)	ListRLAttributesDefinitionsAcrossRuns (1–100 runs)
Environment metric definitions	ListRLEnvironmentMetricDefinitions (optionally some environments), BrowseRLEnvironmentMetricDefinitions (at least one environment)	ListRLEnvironmentMetricDefinitionsAcrossRuns, BrowseRLEnvironmentMetricDefinitionsAcrossRuns (1–100 runs)

The across-runs forms return the union over every environment of every listed run. They take no environment, because environment ids mean nothing across runs.

Per-environment series

ListRLEnvironmentFloatSeriesBuckets (rl-environment-float-series-buckets:list) returns bucketed series for one or more environments of a run — the data behind an environment’s charts:

{
  "workspace": "my-workspace",
  "project": "control",
  "run": "ppo-seed3",
  "environments": ["env-07", "env-11"],
  "scaleSelector": {"episode": "120"},
  "seriesQueries": ["reward(scale=ENVIRONMENT_STEP)"],
  "bucketsCount": "500",
  "useSpecializedRlSeriesBuckets": true
}

scaleSelector chooses the slice: with episode, one episode of each environment, bucketed along its env steps (RlSeriesBucketsOnEpisodeSelector); with rlStep, the points collected at that RL step, bucketed along their order within it (RlSeriesBucketsOnRlStepSelector). Set useSpecializedRlSeriesBuckets — it is required for more than one environment, and every series in the specialized layouts names its environment. The arrays follow the same conventions as step buckets: bucketIndexes lists non-empty buckets and the other arrays align with it.

Aggregating across environments

AggregateRLEnvironmentSeries (rl-environment-series:aggregate) turns one series of every environment into chart lines across an RL step or episode range: at each x, the series is reduced within each environment, then across environments.

{
  "workspace": "my-workspace",
  "project": "control",
  "run": "ppo-seed3",
  "rangeSelector": {"episodeRange": {"fromStep": "0", "toStep": "500"}},
  "seriesExpression": "episode_return(scale=EPISODE):last",
  "aggregations": ["RL_ENVIRONMENTS_AGGREGATION_AVG", "RL_ENVIRONMENTS_AGGREGATION_STD"],
  "filter": "attr(config/difficulty) >= 2",
  "groupings": ["attr(config/map)"],
  "bucketsCount": "250"
}

• rangeSelector — the x axis: an rlStepRange or episodeRange. Open ends resolve to 0 and to the run’s latest value.
• seriesExpression — a selector with one function (first, last, min, max, sum, avg, count or std), reducing the series within each (x, environment) cell. The selector’s scale is part of the series’ identity, not the x axis. On the episode axis the series must be episodic.
• aggregations — one or more reductions across environments: AVG, MIN, MAX, SUM, STD (sample) or MEDIAN. One values array comes back per entry, in order.
• filter — which environments contribute. Series conditions are re-evaluated at every x, and attribute conditions use the value in effect at each x, so the contributing set can change along the axis. attr(environment_id) conditions pick a fixed subset; latest_episode and latest_rl_step are not allowed here.
• groupings — split the output into several lines: by a string attribute (attr(config/map)), by a boolean attribute or series condition, or one line per environment with attr(environment_id) (combine that with a filter selecting a few environments). Environments without a value for a grouping attribute form their own group, reported as unset.
• bucketsCount or bucketWidth — compress the x axis into at most N buckets, or into buckets of a fixed width that line up across runs. Set at most one.

Each returned series has one seriesKeyParts entry per grouping, and aligned arrays: firstSteps (the x positions), counts (the number of environments contributing at each x) and one values array per aggregation. X positions without contributing environments are skipped, so read firstSteps rather than assuming a dense axis.

Because counts always comes back, a 95% confidence band around the mean is avg ± 1.96 × std / sqrt(count) from the AVG and STD arrays.

The server rejects requests that would produce too many output points or series; narrow the range, add buckets or filter environments.