Cost and latency monitoring in LLM systems is the practice of tracking, attributing, and controlling the financial and performance costs of running AI applications in production. Unlike traditional web services where compute costs are relatively predictable, LLM applications have a uniquely volatile cost profile: a single prompt change can double token consumption overnight, an agentic loop can spiral into thousands of API calls, and a misconfigured retry policy can generate a five-figure bill in hours.
The fundamental unit of cost in LLM systems is the token. Every API call consumes input tokens (your prompt) and output tokens (the model’s response), each priced differently per provider and model tier. But tracking raw token counts is only the beginning — the real challenge is attribution: knowing which user, feature, team, or agent session generated the cost, and whether the spend delivered proportional value.
Latency monitoring is equally critical because LLM applications have unique performance characteristics. Users perceive responsiveness through time to first token (TTFT), not total completion time. A streaming response that starts in 200ms but takes 8 seconds to complete feels fast, while a non-streaming response that returns in 4 seconds feels slow. Understanding these nuances is essential for building performant AI applications.
Every LLM API call returns token usage metadata. The foundation of cost monitoring is capturing this data for every call and multiplying by the provider’s per-token pricing:
# Basic cost calculation
cost = (input_tokens * input_price_per_token) +
(output_tokens * output_price_per_token)
# Example: GPT-4 Turbo pricing
# Input: $10.00 / 1M tokens, Output: $30.00 / 1M tokens
# A call with 2,000 input tokens and 500 output tokens:
cost = (2000 * 0.00001) + (500 * 0.00003) = $0.02 + $0.015 = $0.035In practice, you need to track this at the trace level, not just the call level. A single user request in a RAG pipeline might involve an embedding call ($0.0001), a reranker call ($0.001), and a generation call ($0.035) — the total cost is the sum across all spans.
Cost attribution links every token consumed to a business-meaningful entity: user ID, team, feature, session, or agent task. Without attribution, you can see your monthly bill but cannot answer “which feature costs the most?” or “which user is responsible for the spike?” OpenTelemetry trace context propagation is the standard mechanism for threading attribution data through multi-step pipelines.
Production LLM systems implement budgets at multiple granularities to prevent runaway costs:
Budget enforcement can be hard (requests blocked when exceeded) or soft (alerts triggered but requests continue). Most teams use soft caps for user-facing features and hard caps for background/batch processing.
Agentic systems present a unique and dangerous cost profile. A standard chat interaction consumes a predictable number of tokens, but an autonomous agent can amplify costs dramatically:
Agentic systems require 5–30x more tokens per task than a standard chat interaction. A 10-turn research agent can accumulate 500K+ input tokens per task if context is not pruned. Each agent “turn” includes the full conversation history plus new tool results, creating quadratic growth in token consumption. Without per-session budgets and context pruning, a misconfigured agent can consume thousands of dollars in minutes.
# Agentic cost tracking with session budgets
class AgentSession:
def __init__(self, max_budget_usd=5.0):
self.max_budget = max_budget_usd
self.total_cost = 0.0
self.call_count = 0
def track_call(self, input_tokens, output_tokens, model):
cost = calculate_cost(input_tokens, output_tokens, model)
self.total_cost += cost
self.call_count += 1
if self.total_cost >= self.max_budget * 0.9:
alert(f"Agent session at {self.total_cost/self.max_budget:.0%} of budget")
if self.total_cost >= self.max_budget:
raise BudgetExceededError(
f"Session budget ${self.max_budget} exceeded after "
f"{self.call_count} calls (${self.total_cost:.2f})"
)LLM latency is measured at three distinct levels, each revealing different bottlenecks:
The most effective Service Level Indicator (SLI) for LLM applications is the percentage of requests where TTFT stays under a defined threshold, as this aligns closely with perceived responsiveness. A common SLO might be: “95% of requests have TTFT under 800ms.”
Once monitoring reveals where costs concentrate, several optimization strategies become available:
Effective cost dashboards break down spending across multiple dimensions simultaneously: by model (which model tier dominates spend?), by feature (is search or content generation more expensive?), by user segment (are enterprise users consuming proportionally more?), and over time (is cost trending up or down?). Tools like Langfuse provide trace-level cost breakdowns that attribute costs to individual spans within multi-step agent workflows.
A developer changing a system prompt can inadvertently double token usage overnight. Cost monitoring with prompt versioning catches these regressions immediately. By tagging each trace with the prompt version, you can compare average cost per request before and after the change and alert when per-request costs deviate by more than a configurable threshold (e.g., 20%).
Many production systems route all LLM calls through a centralized AI gateway (e.g., Portkey, LiteLLM) that serves as the single point of enforcement for budgets, rate limiting, model routing, and cost tracking. The gateway intercepts every request, checks budget limits, selects the optimal model, and logs token usage — all before the request reaches the provider API.
Organizations running agentic systems at scale are adopting FinOps practices specifically for AI workloads. This includes: forecasting based on historical token consumption patterns, showback/chargeback models that allocate AI costs to business units, and automated anomaly detection that identifies cost spikes from runaway agents within minutes rather than waiting for monthly invoices.