AI Response Streaming (Vercel AI SDK)

Overview

Large language models generate text token by token, and waiting for the entire response to complete before displaying it creates an unacceptable user experience — a blank screen for seconds followed by a wall of text. Streaming solves this by sending each token to the client as it’s generated, so the user sees the response build up in real time, character by character. The Vercel AI SDK provides a complete framework for implementing this pattern: server-side helpers that convert LLM API responses into standardized streams, and React hooks that consume those streams and manage the UI state of messages, loading indicators, and error handling.

The SDK abstracts away the differences between LLM providers — Anthropic’s Claude, OpenAI’s GPT, Google’s Gemini, and others all have different streaming formats (SSE, chunked JSON, proprietary protocols). The AI SDK normalizes these into a single stream protocol based on Server-Sent Events with a defined set of data types: text deltas, tool calls, tool results, finish reasons, and usage metadata. On the server, you call streamText() with your model and prompt; on the client, you call useChat() or useCompletion() and get back reactive state that updates as tokens arrive.

In the platform and its component library, this powers the AI-generated search answers feature. When a user’s search query triggers an AI answer, the backend calls Claude via the AI SDK’s streamText function, and the response streams through a Next.js API route to the React frontend, where useChat renders the answer progressively. The SDK handles token counting for billing, automatic error recovery on network interruptions, and abort signals when the user navigates away mid-stream.

Core Concepts

How It Works

Server-Side API Route

The server creates a streaming endpoint that accepts a message history, calls the LLM, and returns a streaming response. In Next.js App Router, this is a Route Handler.

// app/api/chat/route.ts
import { streamText } from "ai";
import { anthropic } from "@ai-sdk/anthropic";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: anthropic("claude-sonnet-4-20250514"),
    system: `You are a search assistant for an e-commerce platform.
Answer questions about products based on the search results provided.
Keep answers concise and cite specific products when relevant.`,
    messages,
    maxTokens: 1024,
    temperature: 0.3,
  });

  return result.toDataStreamResponse();
}

Client-Side Chat Component

The useChat hook connects to the API route and manages all chat state. The component just renders the messages array and an input form.

import { useChat } from "ai/react";

export function SearchAnswerPanel() {
  const {
    messages,
    input,
    handleInputChange,
    handleSubmit,
    isLoading,
    stop,
    error,
    reload,
  } = useChat({
    api: "/api/chat",
    initialMessages: [],
    onError: (err) => console.error("Stream error:", err),
    onFinish: (message) => {
      // Track token usage for billing
      analytics.track("ai_answer_complete", {
        tokenCount: message.usage?.totalTokens,
      });
    },
  });

  return (
    <div className="answer-panel">
      {messages.map((msg) => (
        <div key={msg.id} className={`message message-${msg.role}`}>
          <div className="message-content">{msg.content}</div>
        </div>
      ))}

      {isLoading && (
        <div className="streaming-indicator">
          <span className="pulse" /> Generating answer...
          <button onClick={stop}>Stop</button>
        </div>
      )}

      {error && (
        <div className="error-banner">
          Something went wrong.
          <button onClick={reload}>Retry</button>
        </div>
      )}

      <form onSubmit={handleSubmit}>
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Ask about search results..."
          disabled={isLoading}
        />
        <button type="submit" disabled={isLoading}>Send</button>
      </form>
    </div>
  );
}

Streaming with Tool Calls

When the AI needs to look up information mid-response, tool calling lets it invoke server-side functions and incorporate the results into the streamed answer.

import { streamText, tool } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import { z } from "zod";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: anthropic("claude-sonnet-4-20250514"),
    messages,
    tools: {
      searchProducts: tool({
        description: "Search the product catalog by query",
        parameters: z.object({
          query: z.string().describe("The search query"),
          limit: z.number().default(5).describe("Max results to return"),
        }),
        execute: async ({ query, limit }) => {
          const results = await productIndex.search(query, { limit });
          return results.map((r) => ({
            name: r.name,
            price: r.price,
            rating: r.rating,
          }));
        },
      }),
      getProductDetails: tool({
        description: "Get detailed information about a specific product",
        parameters: z.object({
          productId: z.string().describe("The product ID"),
        }),
        execute: async ({ productId }) => {
          return await productDB.findById(productId);
        },
      }),
    },
    maxSteps: 3, // Allow up to 3 rounds of tool use
  });

  return result.toDataStreamResponse();
}

Practical Applications

The platform’s AI answer feature uses streaming to deliver real-time answers to merchant search queries. When a shopper searches for “waterproof hiking boots under $150,” the search backend retrieves matching products from the vector index, passes them as context to Claude via the AI SDK, and streams the answer to the storefront widget. The streamed response appears token-by-token in the answer bar, giving the shopper immediate feedback while the full answer generates. Token usage is tracked per-merchant for billing via the onFinish callback’s usage metadata.

The component library exports a reusable StreamingAnswer component that wraps useChat with the design system — loading skeletons, markdown rendering for streamed content, citation formatting for product references, and a “stop generating” button. This component is used across the storefront embed, the merchant preview panel, and the documentation site’s interactive examples. The component handles edge cases like the user navigating away (abort signal), network disconnections (automatic retry with exponential backoff), and rate limiting (queue messages and display a “please wait” indicator).

For the merchant dashboard, a separate useCompletion integration generates natural-language summaries of search analytics: “Your top search this week was ‘red dress’ with 2,340 queries and a 67% click-through rate.” These summaries stream directly into dashboard cards, replacing static text with AI-generated insights that update in real time.

Implementation Patterns

Custom Stream Data

Beyond text, you can send structured data alongside the stream — search result metadata, source citations, or progress indicators that the frontend can render separately from the text content.

import { streamText, createDataStreamResponse } from "ai";

export async function POST(req: Request) {
  const { messages, searchContext } = await req.json();

  return createDataStreamResponse({
    execute: async (dataStream) => {
      // Send search metadata before the LLM response starts
      dataStream.writeData({
        type: "search_context",
        resultCount: searchContext.results.length,
        queryTime: searchContext.queryTimeMs,
      });

      const result = streamText({
        model: anthropic("claude-sonnet-4-20250514"),
        messages,
        onFinish: ({ usage }) => {
          dataStream.writeData({
            type: "usage",
            promptTokens: usage.promptTokens,
            completionTokens: usage.completionTokens,
          });
        },
      });

      result.mergeIntoDataStream(dataStream);
    },
  });
}

Rate Limiting and Queuing

For production deployments, wrap the streaming endpoint with rate limiting to prevent abuse and manage LLM API costs.

import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

const ratelimit = new Ratelimit({
  redis: Redis.fromEnv(),
  limiter: Ratelimit.slidingWindow(10, "1 m"), // 10 requests per minute
});

export async function POST(req: Request) {
  const ip = req.headers.get("x-forwarded-for") ?? "127.0.0.1";
  const { success, limit, remaining } = await ratelimit.limit(ip);

  if (!success) {
    return new Response("Rate limit exceeded", {
      status: 429,
      headers: {
        "X-RateLimit-Limit": limit.toString(),
        "X-RateLimit-Remaining": remaining.toString(),
      },
    });
  }

  // ... proceed with streamText
}

Common Pitfalls

• Buffering the entire stream — If middleware or a reverse proxy buffers the response (e.g., Nginx with proxy_buffering on, or a CDN), tokens accumulate server-side and arrive in bursts instead of streaming smoothly. Ensure your infrastructure passes through chunked responses without buffering.
• Not handling abort signals — When the user navigates away or clicks “stop,” the browser aborts the fetch. If the server doesn’t listen for the abort signal, the LLM call continues running (and billing tokens) even after the client disconnects. The AI SDK handles this automatically, but custom wrappers must forward the AbortSignal.
• Re-rendering on every token — The useChat hook updates state on every token, which means React re-renders on every token. For long responses, this can cause jank. Use React.memo on message components and avoid rendering expensive children (syntax highlighting, markdown) during active streaming — apply formatting only after the stream finishes.
• Missing error boundaries — Network errors during streaming can leave the UI in an inconsistent state (partial message displayed, loading indicator stuck). Always handle the error state from useChat and provide a retry mechanism via reload().
• Ignoring token usage tracking — LLM API calls cost money per token. The onFinish callback provides usage.promptTokens and usage.completionTokens — log these for cost monitoring. Without tracking, a runaway feature can generate unexpected bills.