Tuning & observability

Throughput knobs

These init() arguments control how much the engine buffers and how aggressively it streams:

Argument	Default	Purpose
max_pending_requests	30	In-flight streaming requests — raise it to push more throughput when the backend is the bottleneck.
queue_capacity	10_000	In-process point buffer depth (before backpressure).
batch_max_age_secs	1.0	How long points wait to coalesce into a batch before sending.
max_msg_size	1 << 20	Max serialized request size in bytes.

See the full configuration reference for the rest.

Self-metrics: get_metrics()

metrana.get_metrics() returns a point-in-time snapshot of the engine’s own counters. For each data kind — float_points, rl_float_points, attribute_updates, env_attribute_updates — it reports how much was:

• added — attempted
• enqueued — accepted into the queue
• sent — server-acked over gRPC
• dropped — shed under backpressure or after a channel gave up

With a disk spool configured it also reports disk_persisted (made durable on the spool, not yet seen by the backend) and the recovery counters recovered_spool_segments / recovered_spool_messages (drained from the spool back to the backend), alongside transport/health counters (connection_attempts, requests_sent, send_errors).

Did anything get lost?

Comparing added vs sent tells you whether anything was lost:

m = metrana.get_metrics()
print(m.float_points_added, m.float_points_sent, m.float_points_dropped)

Rates over a window

The counters are monotonic, so diff two snapshots over a window to get rates — useful for a periodic health log:

import time

prev = metrana.get_metrics()
time.sleep(10)
now = metrana.get_metrics()

sent_per_sec = (now.float_points_sent - prev.float_points_sent) / 10
backlog = now.float_points_added - now.float_points_sent      # added but not yet acked
if now.float_points_dropped > prev.float_points_dropped:
    print("data loss in the last window — enable a disk spool, or raise queue_capacity / retries")

The same snapshot is returned inside CloseInfo.metrics at the end of a run.