Skip to content
Metrana Docs

Agentic traces

A trace records one complete agent invocation — in post-training, one rollout. It is a tree of spans: thoughts, loop iterations, LLM calls, tool calls, sub-agents. The trace itself is the root span (span 0) and every other span hangs off it. Each span can carry payloads: the prompt and completion of an LLM call, a tool’s input and output, the agent’s thoughts, attributes.

Traces are written with UpdateTraces — gRPC, or POST /api/ingest/v1/traces:update — or over the long-lived UpdateTracesStream (see Streaming), and read with the Query API’s trace RPCs.

Anchoring

Section titled “Anchoring”

A request names where its traces belong:

  • To a run — set runName. Every trace then records the training step whose policy produced it, so rollouts line up with training progress.
  • To the project alone — leave runName unset, for inference traces that belong to no training run. The project must exist.
{
  "workspace": "my-workspace",
  "project": "agents",
  "runName": "grpo-7b-v2",
  "loggerId": "5813377020184215",
  "traceUpdates": [ … ]
}

loggerId identifies the writing process; use a stable non-zero value per process. traceUpdates are applied in order; an empty list is a no-op that still resolves the run and reports your write limit, which is what a stream’s opening message and idle keepalives send.

Trace updates

Section titled “Trace updates”

Each TraceUpdate carries the trace’s identity and one event: traceOpen, spanOpen, spanClose or traceClose. A span’s open must come before its close — in the same request, or an earlier one.

Identity

Section titled “Identity”

Every update restates the full TraceIdentity, unchanged, for the life of the trace:

Field Rule
clientTraceKey 16 random bytes you generate when the trace starts (base64 in JSON). Must be random, never a counter: two traces with the same key become one.
traceStartTime When the trace started. Required on every update; it is part of the trace’s identity, so never re-read the clock for it.
step The training step of the generating policy. Required when anchored to a run; must be unset otherwise.
runId Leave empty when runName is set — the service fills it in.

Opening and closing

Section titled “Opening and closing”
"traceUpdates": [
  {"identity": {…}, "traceOpen": {
    "name": "math-solver", "kind": "SPAN_KIND_SUB_AGENT",
    "payloads": [{"dataOffset": 0, "dataLength": 12, "taskInput": "V2hhdCBpcyAyKzI/"}]
  }},
  {"identity": {…}, "spanOpen": {
    "spanId": "1", "parentSpanId": "0", "ordinal": 0, "kind": "SPAN_KIND_THOUGHT", "name": "plan",
    "startTime": "2026-09-30T09:00:00.120Z",
    "payloads": [{"dataOffset": 0, "dataLength": 25, "thought": "SSBzaG91bGQgYWRkIHRoZSBudW1iZXJzLg=="}]
  }},
  {"identity": {…}, "spanClose": {
    "span": {"spanId": "1", "parentSpanId": "0", "ordinal": 0, "kind": "SPAN_KIND_THOUGHT", "name": "plan",
             "startTime": "2026-09-30T09:00:00.120Z"},
    "endTime": "2026-09-30T09:00:00.480Z", "status": "SPAN_STATUS_OK", "tokenCount": "18"
  }},
  {"identity": {…}, "traceClose": {
    "open": {"name": "math-solver", "kind": "SPAN_KIND_SUB_AGENT"},
    "endTime": "2026-09-30T09:00:01.050Z", "status": "SPAN_STATUS_OK",
    "spanCount": 1, "iterationCount": 0, "tokenCount": "18",
    "payloads": [{"dataOffset": 0, "dataLength": 16, "finalOutput": "VGhlIGFuc3dlciBpcyA0Lg=="}]
  }}
]
  • traceOpen starts the trace: its name (the agent or task, which you search and list traces by), its kind — any kind may root a trace: SUB_AGENT for an agent rollout, LLM_CALL for an inference trace that is one model call, FUNCTION for a plain function — and optionally the task input, the input body of the root’s own kind, and the trace’s attributes.
  • spanOpen starts a span. spanId is a counter you increment from 1 within each trace; parentSpanId is another span’s id, or 0 for the root; ordinal orders siblings. name (up to 512 bytes) and startTime are required.
  • spanClose restates the span’s open exactly as sent — without its payloads — then adds endTime, a terminal status and the span’s tokenCount. Restating keeps the stored span complete, because the close replaces the open’s record.
  • traceClose likewise restates traceOpen (without payloads), and records the outcome and the totals shown in trace lists: spanCount, iterationCount and tokenCount.

A terminal status is OK, ERROR, TRUNCATED (the work finished but its payload was capped) or CANCELLED; RUNNING is what an open span shows until its close arrives.

Span kind Represents
SPAN_KIND_THOUGHT The agent’s reasoning.
SPAN_KIND_ITERATION One pass of the agent’s perceive-act loop (not a training step).
SPAN_KIND_LLM_CALL A call to a model.
SPAN_KIND_TOOL_CALL A tool invocation.
SPAN_KIND_SUB_AGENT A nested agent; its work appears as its child spans.
SPAN_KIND_ARTIFACT An artifact the agent produced.
SPAN_KIND_FUNCTION A plain function call — the kind-neutral node a decorator wraps any function in.

Payloads

Section titled “Payloads”

A SpanPayload carries one body, or one chunk of it. Which kinds a node may carry follows what it is:

Belongs to Payload kinds
Any node, root or span attributes, and the generic input / output — what a function was called with and returned, beside the kind-specific bodies
The trace (traceOpen, traceClose) taskInput (on open), finalOutput (on close), plus the bodies of the root’s own kind: an LLM_CALL root takes llmPrompt and llmCompletion, a TOOL_CALL root toolInput and toolOutput, a THOUGHT root thought
A span (spanOpen, spanClose) thought, toolInput, toolOutput, llmPrompt, llmCompletion

Storage keeps one body per span and kind, written once: sending a second llmPrompt for the same span is not supported, and reading that body back afterwards may return a mix of the two or stop short. Re-sending the same body — a retry after a reconnect — is harmless. Send what is known at open on the open — a prompt, a tool’s input, the task — and what the node produced on the close, so a running span already shows what it was asked.

  • Text bodies are UTF-8 bytes — base64-encoded in JSON. Invalid UTF-8 is rejected.
  • attributes is a set of typed Attributes: at most 256 per payload, each value up to 64 KiB, names unique. Names matching the span fields the Query API filters on (such as op_name, step, status, kind, token_count) or lists spans by (run, trace_id, span_id, parent_span_id, ordinal) are reserved.
  • dataLength is the size of the whole payload — bytes of text, or number of attributes — and is required on every chunk. dataOffset is where this chunk starts.

Large bodies. Split a body that will not fit one request into chunks: send each with its own dataOffset and the same dataLength. Chunks may arrive in any order and across any number of requests, but two chunks of one kind in a set must not share an offset, and no chunk may extend past dataLength. Bodies are capped at 16 MiB; anything beyond is cut rather than rejected — a trace should never fail your training job — and readers see that it was truncated.

Streaming

Section titled “Streaming”

A client that traces continuously keeps one UpdateTracesStream open instead of a request per batch. It is a bidirectional gRPC stream of the same UpdateTracesRequest messages, with one UpdateTracesResponse per message, in order:

  • Open it with a priming message — a request with empty traceUpdates. It is acknowledged after the workspace check alone, so an unauthorized stream is refused at open rather than at its first data. The same content-free message serves as an idle keepalive.
  • Messages are delivered in the order sent. The service pipelines a stream’s messages — validation and lookups overlap — but hands their updates to storage strictly in message order, so a span’s close can never overtake its open. Order is all a client must keep: send a span’s open before its close.
  • A failed message ends the stream with its error as the stream’s status. Messages after it may or may not have been written; none is acknowledged. The client reconnects, primes again and re-sends from its first unacknowledged message. Re-sent updates are harmless (see Errors).
  • Requests may be zstd-compressed (grpc-encoding: zstd); the service accepts it on every RPC. Trace bodies are text and shrink several-fold, so a streaming client should compress.

The Python SDK’s metrana.tracing uses this stream.

Errors

Section titled “Errors”

Validation failures are INVALID_ARGUMENT (400) naming the offending update. An unknown run is NOT_FOUND (404). UNAVAILABLE (503) is transient: retry with backoff. RESOURCE_EXHAUSTED (429) means the request did not fit under your workspace’s or organization’s write rate: back off for longer and send less per second. Re-sent updates are harmless, because each event’s record is replaced by later writes for the same span.