Reinforcement learning data

Reinforcement-learning runs log data per environment, on a two-level step: a major RL step (training progress) and a minor step within it. Read Reinforcement learning concepts first if the terms are new.

RL data travels in the same requests as everything else — rlSeries and envAttrUpdates on an UpdateRuns run update, or rlSeries / envAttrUpdates on a stream message.

Which series type to use

What you log	Message	Scale	Minor step
Per-update training metrics (policy loss, entropy, …)	SeriesUpdate	ML_STEP (default)	— the step is the RL step
Per-episode metrics (episode return, length, …)	RLSeriesUpdate	EPISODE	the episode number
Per-environment-step metrics (reward, action values, …)	RLSeriesUpdate	ENVIRONMENT_STEP	a counter within the RL step, or your own env-step numbers

Per-update metrics are ordinary series — see Logging metrics and attributes. The rest of this page covers the two RL scales.

RL series

An RLSeriesUpdate is identified like any series — metric name, scale and labels — plus the environment it belongs to:

"rlSeries": [{
  "metricName": "reward",
  "environmentId": "env-07",
  "scale": "ENVIRONMENT_STEP",
  "ordering": "ORDERING_MODE_ABSOLUTE_ORDERED",
  "rlStepBatches": [
    {"rlStep": "120", "episode": "41", "firstMinorStep": "0", "values": [0.0, 0.0, 1.0], "timestampMillis": "1759233600000"}
  ]
}]

• environmentId is required. It appears in portal URLs, so it follows the name rules, up to 128 bytes. Avoid /: it is currently rewritten to -.
• ordering follows the same rules as for float series (ABSOLUTE_ORDERED or ABSOLUTE_UNORDERED) but governs the minor step. The RL step itself must never decrease within a series; points behind it are deduplicated.
• rlStepBatches holds one RlStepBatch per (rlStep, episode) pair contributing to this series.

The older points, floatPoints and top-level episode fields are deprecated; use rlStepBatches.

RlStepBatch

A batch shares one RL step and, optionally, one episode across all its points, and is columnar like a float PointBatch:

Field	Meaning
rlStep	The RL step of every point in the batch.
episode	The episode of every point in the batch, when the series is episodic.
values / valuesF32	Point values; exactly one of them (double or float precision).
firstMinorStep / minorSteps	Consecutive minor steps starting at firstMinorStep, or one per value.
timestampMillis / timestampsMillis	One timestamp for every point, or one per value.
treatMinorsAsFlatSteps	The minor steps are your own environment-step numbers (see below).
envStepsFromOne	Number an episode’s environment steps from 1 instead of 0 (see below).

A batch holds at most 10,000 values.

Per-episode metrics

On the EPISODE scale, put the episode number in the minor step and leave episode unset:

{
  "metricName": "episode_return",
  "environmentId": "env-07",
  "scale": "EPISODE",
  "ordering": "ORDERING_MODE_ABSOLUTE_ORDERED",
  "rlStepBatches": [{"rlStep": "120", "firstMinorStep": "41", "values": [212.5]}]
}

Use ABSOLUTE_ORDERED when episodes arrive in increasing order from one writer; use ABSOLUTE_UNORDERED when several writers log the same series or episodes can arrive out of order.

Per-environment-step metrics

The ENVIRONMENT_STEP scale records every step an environment takes. Each point has an RL step, an optional episode, and a position on the environment’s env-step axis, which counts up across RL steps and — for an episodic series — restarts with each new episode.

Whether a series is episodic is fixed by its first write: set episode on every batch, or on none.

There are two ways to fill the env-step axis.

Let Metrana count (recommended). Send minor steps that count points within the RL step — 0, 1, 2, …, restarting at 0 for each new RL step — and Metrana accumulates the env-step axis from them. There is no client-side cursor to keep, so this survives process restarts. Episodic axes start at 0; set envStepsFromOne on every batch of the series to start them at 1 (requires ABSOLUTE_ORDERED and an episode).

{"rlStep": "121", "episode": "41", "firstMinorStep": "0", "values": [0.0, 0.5]}

Number the steps yourself. Set treatMinorsAsFlatSteps and send your environment’s own step numbers as the minor steps; they are stored as given. Requires ABSOLUTE_ORDERED. On an episodic series, dropping back to step 0 or 1 within an episode restarts that episode, discarding the steps it had recorded.

{"rlStep": "121", "episode": "41", "minorSteps": ["1012", "1013"], "values": [0.0, 0.5], "treatMinorsAsFlatSteps": true}

Keep one approach for the lifetime of a series.

Environment attributes

Environments carry their own configuration — map, difficulty, seed — as attributes, versioned by episode. Each EnvAttrUpdates sets attributes for one environment:

"envAttrUpdates": [{
  "environmentId": "env-07",
  "episode": "41",
  "attributes": [
    {"path": "config/map", "stringValue": "maze-large"},
    {"path": "config/difficulty", "intValue": "3"}
  ]
}]

• Attributes logged at an episode stay in effect for later episodes until the same path is logged again. Without episode, they apply from the environment’s start.
• Values use the same AttributeUpdate types and limits as run attributes. There is no sys/ namespace here, but the paths environment_id, latest_episode and latest_rl_step are reserved.
• The Query API reads them as of any episode — see RL environments.