Training Hub › Tracing & Cost Monitoring

LLM Tracing & Observability

Observability & Operations advanced Building with AI 12 min read

Overview

LLM Tracing and Observability is the practice of instrumenting every step of an AI application — from the initial user query through vector search, tool execution, prompt construction, and generation — to produce structured logs that enable debugging, performance analysis, and quality monitoring. Unlike traditional application monitoring where you track HTTP status codes and response times, LLM observability must capture semantic dimensions: what was in the prompt, how many tokens were consumed, which retrieved chunks were used, and whether the output was faithful to the source material.

In production AI systems, failures are rarely binary. A RAG pipeline doesn’t crash with a 500 error — it silently returns a plausible-sounding but wrong answer. Without tracing, you have no way to determine whether the problem originated in retrieval (wrong chunks fetched), prompt construction (relevant data lost in compression), or generation (model hallucinated despite good context). Observability transforms these invisible failures into diagnosable events.

The stakes are amplified in agentic systems where a single user request can trigger dozens of LLM calls, tool invocations, and retrieval operations across multiple turns. A 10-step research agent might make 30+ LLM calls, each dependent on prior outputs. Without structured tracing that links these calls into a coherent execution tree, debugging a bad final answer becomes effectively impossible.

How It Works

Traces, Spans, and the Execution Tree

LLM observability borrows heavily from distributed tracing concepts, particularly OpenTelemetry. The fundamental unit is a trace — a complete record of a single user request flowing through your system. Each trace is composed of spans, which represent individual operations: an embedding call, a vector search, an LLM generation, a tool execution.

Key Concept: Trace Hierarchy

A trace is a tree of spans. The root span represents the entire request. Child spans represent sub-operations. For example: User Query (root) → Embedding Generation → Vector Search → Reranking → Prompt Construction → LLM Generation. Each span records its start time, end time, status, and custom attributes like token counts and model name.

In agentic workflows, spans can nest deeply. A root “agent_run” span might contain child spans for “planning”, “tool_call” (which itself contains “api_request”), “reflection”, and “final_generation”. This hierarchical structure lets you see exactly where time and tokens are being spent.

OpenTelemetry GenAI Semantic Conventions

OpenTelemetry has established semantic conventions specifically for generative AI operations. These conventions standardize the attribute names attached to spans, making traces interoperable across tools and vendors. Key attributes include:

These conventions replaced the earlier deprecated gen_ai.prompt and gen_ai.completion attributes (removed in v1.38.0), moving to a message-structured model that better represents multi-turn conversations and tool-use patterns.

Instrumentation Approaches

There are two primary approaches to instrumenting LLM applications:

Auto-instrumentation wraps SDK calls automatically. Libraries like OpenLLMetry (by Traceloop) and Langfuse’s integrations can patch OpenAI, Anthropic, and other SDK clients to emit spans without code changes:

# Auto-instrumentation with Langfuse + OpenAI
from langfuse.openai import openai  # patched import

# All calls automatically traced
response = openai.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain RAG"}]
)
# Langfuse automatically captures: model, tokens, latency, messages

Manual instrumentation gives you fine-grained control over what gets traced. You create spans explicitly around custom logic:

from langfuse import Langfuse

langfuse = Langfuse()

# Create a trace for the entire request
trace = langfuse.trace(name="rag-pipeline", user_id="user-123")

# Span for retrieval
retrieval_span = trace.span(name="vector-search")
results = vector_db.search(query_embedding, top_k=10)
retrieval_span.end(output={"num_results": len(results)})

# Span for generation
generation = trace.generation(
    name="answer-generation",
    model="claude-3-sonnet",
    input=prompt,
    output=response.text,
    usage={"input": 1500, "output": 300}
)

Key Observability Platforms

The LLM observability landscape has matured significantly:

What to Capture at Each Stage

A complete observability setup captures data at every stage of the pipeline:

  1. Query Processing — Original query, rewritten queries, detected intent, extracted entities
  2. Retrieval — Search latency, number of results, similarity scores, chunk IDs retrieved
  3. Reranking — Reranker model, latency, score changes, documents promoted/demoted
  4. Prompt Construction — Final prompt token count, system prompt version, number of chunks injected
  5. Generation — Model name, temperature, input/output tokens, latency (TTFT and total), finish reason
  6. Tool Calls — Tool name, arguments, return value, execution time, success/failure
  7. Post-processing — Output validation results, guardrail checks, citation extraction

Practical Applications

Debugging Silent Failures

Consider a RAG system where users report that answers about a recently updated product are still referencing old specifications. With tracing, you can examine the retrieval span and discover that the vector search is returning stale chunks because the embedding pipeline hasn’t reprocessed the updated documents. Without tracing, this appears as “the AI is wrong” with no clear path to diagnosis.

Performance Optimization

Traces reveal exactly where latency accumulates. A typical finding: retrieval takes 200ms, reranking takes 400ms, and LLM generation takes 2.5 seconds — but prompt construction (serializing chunks into the prompt) takes 800ms due to an inefficient template engine. This kind of bottleneck is invisible without span-level timing.

Quality Monitoring with LLM-as-Judge

Platforms like Langfuse support running automated evaluations on traced generations. You can configure an “LLM-as-judge” that scores every response for faithfulness, relevance, and helpfulness. These scores are attached to traces, enabling you to build dashboards that track quality metrics over time and alert when scores drop below thresholds.

Multi-Turn Conversation Debugging

Modern observability platforms support multi-turn conversation tracing, linking individual turn traces into a session-level view. This is critical for debugging issues like context window overflow (where earlier turns get truncated), conversation state corruption, or memory retrieval failures in systems with long-term memory.

A/B Testing and Prompt Versioning

By tagging traces with prompt versions and model configurations, you can compare performance across variants. Langfuse’s prompt management system versions prompts and links generations back to specific versions, making it straightforward to determine whether a prompt change improved or degraded quality.

Key Takeaways

Further Reading