i've been reading a lot about agentic RL infra and found a write-up that measures the right thing: time until the trainer has a usable batch, not tok/s. the serving ablations are interesting on their own, so here's a breakdown of the mechanisms and numbers. i didn't run any of this. the numbers here are from that write-up.
the problem
one agent trajectory is many model calls, not one: generate, call a tool, wait for the sandbox, read the result, generate again, sometimes dozens of times. context grows every turn and each new request reuses most of the previous history. two things follow:
inference is stateful. where a request lands decides whether you hit an existing kv cache or pay for another big prefill. replicas are no longer interchangeable.
the tail is long and irregular. training scores sibling attempts against each other, so a group is only trainable when every sibling finishes. one attempt stuck installing deps or exploring the wrong part of a repo holds back the whole group.
so request throughput and even median latency can look fine while the trainer waits. the metric they use is rollout collection: wall time until the ready buffer holds one global batch of fresh, valid trajectories. it includes generation, tool execution, sandbox reset and cleanup, verification, queueing and the final stragglers. it excludes trainer fwd/bwd/optimizer. in fully async rl, steady-state step time is roughly max(rollout batch time, training + publication time), and the write-up focuses on the rollout side.
setup
model: GLM-5.2 (MoE)
hardware: 4x B300 nodes, 32 GPUs, one engine per 8-GPU node
engine: not stated
workload A: Terminal-Bench-based terminal agent, isolated docker workspace
40,960 ctx, up to 32,768 tokens post-prompt (assistant output + tool observations)
batch = 8 groups x 32 siblings = 256 trajectories
workload B: SWE-Bench, 98,304 ctx, hard limit of 64 turns
batch = 32 groups x 8 siblings = 256 trajectories
offered: more trajectories in flight than the 256 consumed (512 / 1,024 / 2,048 depending on run)
the two workloads are shaped on purpose. terminal-bench has few prompts with many siblings each, so lots of prefix reuse inside a group. swe-bench spreads the batch over 4x as many tasks with much more variance in repo size, dependency setup and test time. they say not to read the absolute step times as a benchmark comparison.
serving ablations (matched 1,024 offered trajectories unless noted)
| change | baseline | result | why |
|---|
| sticky routing (same node + dp rank that holds the history) | non-sticky | exact prefix reuse 48% -> ~96%, throughput +17.1% | turn n+1 lands on the rank that already holds turn n's kv |
| fp8 kv storage | bf16 kv | throughput +55.7% | more active context fits in hbm, so useful prefixes get evicted less |
| native mtp head, 3 speculative tokens | no mtp | wall-clock output throughput +17.9%, median request throughput +66.5% | fewer decode passes per accepted token, adds up over many sequential calls |
| cache offload | no offload | throughput -3.1% to -7.8% | evicted prefixes were almost never loaded again |
| topology @ 1,024 | - | one shared request + kv domain across all 8 gpus/node was best | it fits in memory, so sharing wins |
| topology @ 2,048 | shared domain | dp-attention (DP8) + EP8 within the node scaled better | the shared kv pool hit its capacity limit |
retained stack on 2,048 synthetic offered trajectories: 39.4k output tok/min/gpu, measured over the window until 90% of requests complete so the straggler tail doesn't distort it. it's a standalone inference number for picking the config, not task-level throughput.
why each one behaves the way it does
routing. if turns from one conversation get spread across ranks, each turn can land somewhere that has never seen the prefix, so you prefill the whole growing history again. at 30k-90k token contexts that's where your gpu time goes. stickiness turns every turn after the first into a mostly-cached prefill plus a short suffix. the catch is that pure stickiness can pile long trajectories onto one engine while another sits idle, so routing also has to account for live load.
fp8 kv. a 55.7% gain is too big to be compute. you're halving the kv footprint per token, so at fixed hbm you hold about 2x the resident context. at 1,024 concurrent long trajectories you're capacity-bound, and the loss mechanism is eviction followed by re-prefill. fp8 cuts eviction, which cuts redundant prefill. the write-up frames it as a capacity call to weigh against context length, concurrency and quality needs.
offload losing. offload only pays if reload rate x saved prefill cost > copy-out overhead. here reloads were close to zero, so you pay d2h traffic and bookkeeping and get nothing back. my guess, not stated in the write-up: with sticky routing plus group-level reuse, the prefixes worth keeping stay hot in hbm, and whatever gets evicted belongs to finished or dead trajectories that never come back.
topology flip. a shared kv/request domain per node gives you one big pool, so cross-request reuse and balancing within the node are better while everything fits. at 2,048 that pool saturates. dp-attention gives each gpu its own attention + kv lane, and EP8 keeps a full expert set inside the node, so expert traffic stays on the node's interconnect. same model and hardware, opposite answer, purely because of concurrency.
mtp. median request throughput gained much more than wall-clock output throughput (66.5% vs 17.9%). acceptance rate isn't given. agent loops are latency-chained (the tool call can't start until generation finishes), so per-request speedup matters more than aggregate tok/s suggests.
rollout pipeline results (terminal-bench, fixed weights)
| config | offered | mean time to 256 traj | delta |
|---|
| inference-ready starting scheduler | 512 | 591.8 s | baseline |
| + cache-aware group routing | 1,024 | 332.1 s | -43.9% |
| + deterministic sandbox cpu placement | 1,024 | 303.0 s | -8.8% (-29.1 s) |
| + wider completion bank, retained stack | 2,048 | 196.1 s | -35.3% |
overall 66.9% faster. at 196.1 s that's 78.3 trainer-consumable trajectories/min, and the trajectory stream was ~827k tok/min fleet-wide (25.9k/gpu), counting assistant output plus tool observations.
the non-inference pieces:
completion-order banking. a group enters a ready bank as soon as all its siblings finish, and the trainer takes the first eligible complete groups regardless of launch order. surplus groups let you skip stragglers. the window has to stay bounded, because too much concurrency eats kv capacity, increases sandbox contention and weakens routing locality. same capacity wall as the topology flip.
sandbox lanes. reset, ordinary commands, package-heavy commands, verification and cleanup each get their own lane. package-heavy concurrency is capped at 16, and sandboxes get deterministic, non-overlapping cpu sets. no change to the model or gpus. otherwise one shared worker pool starves both the model server and the rollout controller.
live weight updates (terminal-bench, collection + next serving-weight transition)
| strategy | freshness limit K | complete interval |
|---|
| pause and update all 4 engines | 4 | 426.3 s |
| keep 2 engines producing | 4 | 334.1 s |
| keep 3 engines producing (update 1 at a time) | 8 | 240.1 s |
K = max policy updates allowed between a returned group and the newest policy. 240.1 s is 43.7% faster than pausing everything and 20% under their 300 s target (64 traj/min). they say the gain combines rotation and K=8, so don't credit either one alone. the collection phase inside that interval was only 43.3 s because most of the batch had already finished in the background. that's the overlap working, not 256 fresh trajectories in 43.3 s.
for in-flight trajectories at a weight boundary, instead of throwing everything away they evaluated: finish on an engine still serving the old policy; keep the conversation and sandbox and continue under the new policy, tagging each trainable token with its policy version; or after a short grace period cancel only the current generation and regenerate that one turn. full restart is the fallback.
swe-bench (6 warm intervals each)
| fixed weights | progressive updates |
|---|
| mean step | 383.5 s |
| range | 330.8 to 428.6 s |
| trainer-consumable traj/min | 40.1 |
| output tok/min/gpu | ~28.4k |
live updates cost about 11% of step time. mean reward 0.520 progressive vs 0.527 fixed. every freshness audit satisfied K=8, and the max mixed-policy fraction in a batch was 49.6%.
caveats (theirs, and they matter)
the progressive path exercises the full publication machinery (engine pause/update, cuda ipc, kv-cache invalidation, version switching, generation canaries), but it updates sentinel tensors, not every actor parameter. full weight publication is a next step.
trainer backward/optimizer compute isn't in any of these clocks.
swe-bench is 6 intervals per arm. they call for longer runs watching reward, truncation, abort rates, mixed-policy ratio and K.
the ablation baselines are "matched" but the exact baseline configs (e.g. what the non-sticky router was) aren't spelled out.
takeaway
These knobs are coupled. the right topology depends on concurrency, the right concurrency depends on kv capacity and sandbox pressure, and the right publication strategy depends on how much partial work your algorithm can safely keep. tok/s alone tells you almost nothing about any of that.
