Reinforcement learning concepts

Reinforcement learning (RL) runs log more than a single training curve: they involve environments that produce episodes and per-step data, all tied back to training progress. This page explains the concepts so the Environments views make sense.

Core concepts

These nest from largest to smallest:

• An environment is a single instance of the world an agent interacts with during an RL run, identified by an env_id. A run can have many environments running in parallel.
• An episode is one run-through within an environment, indexed by an episode number. An episode is a sequence of environment steps from reset until termination, and produces summary metrics such as episode_reward and episode_length.
• An environment step is one transition in the environment — a single state → action → reward step. This is the smallest unit; episodes are made up of many environment steps.

Cutting across this nesting is the training axis:

• An RL step is a batch of environment steps collected with the same policy during training; in practice it is a gradient-update counter for “how far training has progressed”. A single RL step can span partial episodes and multiple environments, so it groups environment steps by when in training they were collected rather than by which episode they belong to. (This framing gets fuzzier for offline or asynchronous training methods.)

The easiest way to see how these fit together is from the bottom up. The lowest-level thing you log is an environment step — one transition. Each environment step is tagged with the episode it belongs to (how many episodes that environment has completed) and the RL step at which the experience was collected (how far into training you were). Episodes and RL steps therefore give you two ways to line up and aggregate the same underlying steps.

Scales

Each of these concepts maps to a scale — the dimension a logged value is indexed by:

• RL step (ML_STEP) — indexed by training gradient-update step.
• Episode — indexed by episode boundary within an environment.
• Environment step (ENV_STEP) — indexed per step within an episode (high-frequency).

Standard supervised metrics use the ordinary step scale instead — see Metrics, series & attributes.

How logging maps to the portal

The values your code logs land in the Environments views like this:

What you log	Scale	Where it shows up
Per-gradient-update RL metrics	RL step (ML_STEP)	Charts, scrubbed by RL step
Per-episode metrics (with env_id)	Episode	The environment’s episode number and episode charts
Per-environment-step metrics (with env_id)	Environment step	High-frequency environment charts

The RL step is the common thread: it lets you line up an environment’s episodes against the training step at which they happened.

Evaluation vs. training

Metrics logged as evaluation are kept distinct from training metrics so you can tell them apart when analyzing — the same idea as for supervised runs.

Next

• Environments overview — the table of environments in a run.
• Episodes & steps — navigating by RL step or episode.
• Environment charts — visualizing a single environment.