Multi-step LLM agents produce valuable experience while they run: prompts, tool calls, retrieved context, intermediate outputs, rewards, failures, retries, and final answers. The hard part is turning that messy execution history into training data without rewriting the agent as a special reinforcement-learning environment. Span-based agent RL treats the running agent as the source of telemetry and converts selected spans into transition records that an optimizer can consume.
Agent Lightning is a concrete example of this pattern. Its repository and paper describe a training-agent disaggregation architecture: the agent runner keeps executing normal application code, while a separate algorithm process reads traces, computes credit assignment, and publishes updated resources such as prompt templates or model weights. The agent does not have to be flattened into one giant sequence; each LLM call can become a short transition with its own input, output, context, and reward.
The transferable idea is not the package name. It is the system boundary: instrument the agent like production software, persist ordered spans in a store, map task outcomes back onto LLM-call transitions, and train through a separate control plane. This makes reinforcement learning practical for agents built with existing orchestration frameworks, multi-agent workflows, retrieval systems, and tool-use loops.
Traditional agent RL often couples rollout execution, model inference, reward handling, and optimization into one code path. Disaggregation splits those responsibilities. Runners execute tasks on CPU-friendly infrastructure and emit spans. Algorithms train models on GPU-friendly infrastructure and enqueue new tasks or resources. A shared store is the contract between both sides.
A span is a structured record of work. For agent training, useful spans can represent LLM calls, tool invocations, graph edges, reward emissions, or custom application blocks. The store assigns sequence identifiers so downstream adapters can reconstruct the execution even when clocks drift or multiple runners write concurrently.
When a task succeeds or fails after many LLM calls, the final reward is too coarse for direct training. LightningRL-style credit assignment decomposes a full trajectory into per-call training transitions. Each transition can then be fed to ordinary single-step RL algorithms such as PPO or GRPO, keeping sequence lengths shorter and isolating the model-generated tokens that should receive gradient signal.
If every agent action is captured as ordered telemetry, the training system can evolve independently from the application. The runner only needs to expose what happened and what changed; the algorithm decides which spans are trainable, how rewards are assigned, and which resources should be updated.
The runner claims a rollout, executes the agent, and writes spans into a central store. Each span is associated with a rollout and attempt, then receives a monotonic sequence number. That ordering matters because distributed traces may arrive out of wall-clock order.
async def run_rollout(store, tracer, rollout):
attempt = await store.create_attempt(rollout.rollout_id)
async with tracer.trace_context(
name="agent-rollout",
store=store,
rollout_id=rollout.rollout_id,
attempt_id=attempt.attempt_id,
):
result = await agent.solve(rollout.input)
await emit_reward(score=result.score, reason=result.verdict)
await store.mark_succeeded(rollout.rollout_id, attempt.attempt_id)An adapter reads the span tree and selects model actions. For each LLM call, it builds a transition containing the prompt, generation, relevant context, action metadata, and reward. Tool spans are usually context, not directly trained tokens, but they help explain why a later model call deserves positive or negative credit.
def transitions_from_trace(spans):
final_reward = find_final_reward(spans)
for span in ordered(spans):
if span.kind == "llm.call":
local_reward = assign_credit(span, spans, final_reward)
yield {
"prompt": span.attributes["llm.prompt"],
"completion": span.attributes["llm.completion"],
"reward": local_reward,
"rollout_id": span.rollout_id,
}The algorithm consumes transitions, updates prompts or weights, and writes a versioned resource snapshot back to the store. Future runners fetch the latest compatible resource before executing new work. This closes the learning loop without requiring the agent framework to know how PPO, GRPO, supervised fine-tuning, or prompt optimization is implemented.
A text-to-SQL system may contain separate agents for generation, checking, and repair. Span-based RL can optimize only the LLM calls that control SQL generation or rewriting while leaving deterministic database execution and validation code untouched. Rewards can combine executable-query checks, answer correctness, and safety filters.
For multi-hop retrieval tasks, the agent’s search-query generation and answer synthesis steps can be trained separately. Retrieval spans provide evidence about which documents were available, while LLM-call spans expose whether the model asked useful queries and used retrieved evidence correctly.
Math or code agents often fail by calling a tool too early, too late, or with malformed arguments. A span-based interface lets the reward function score the final answer while the credit assignment module gives local feedback to the decisions that selected tools and integrated results.
Use the store as the single coordination primitive for rollouts, attempts, spans, resources, and worker liveness. Runners poll work and write traces. Algorithms enqueue work and read traces. Versioned resources make it possible to compare model or prompt versions across repeated evaluations.
class TrainingStore:
async def enqueue_rollout(self, task): ...
async def dequeue_rollout(self, worker_id): ...
async def next_span_sequence(self, rollout_id, attempt_id): ...
async def add_span(self, span): ...
async def update_resources(self, resources): ...
async def latest_resources(self): ...Define stable semantic fields for prompts, completions, tool names, reward values, model identifiers, resource versions, and parent-child relationships. Inconsistent trace attributes force every adapter to recover meaning from ad hoc payloads, which defeats the purpose of using telemetry as the training interface.
Keep training code out of the production agent loop. The runner should be able to execute a task and emit spans even when no optimizer is attached. The optimizer should be able to replay stored traces, test new reward functions, and run offline experiments before publishing updated resources.