Chat Agent with Multi-Turn Conversational Memory

Overview

Multi-turn conversational agents must maintain coherent context across extended dialogues, remembering what the user said five messages ago while managing the finite context window available to the underlying language model. Unlike single-shot question-answering, a chat agent accumulates message history, tracks conversational state, resolves coreferences (“what about that other one?”), and decides when to invoke tools — all while streaming responses back to the user in real time.

In a typical TypeScript chat agent implementation, the agent handles multi-turn conversations by maintaining a structured message history, supporting tool-calling with streaming callbacks, and implementing clarification request logic for ambiguous queries. The system must balance comprehensive context retention against token budget constraints, deciding which messages to keep verbatim, which to summarize, and which to drop entirely.

This topic covers the architecture of a production chat agent: message accumulation strategies, context window management, tool-calling integration with streaming output, and handling ambiguous user input through clarification requests. These patterns apply broadly to any system where an LLM must maintain conversational continuity across multiple exchanges.

Core Concepts

How It Works

The chat agent operates as a loop: receive a user message, construct the prompt from accumulated history plus system instructions, call the LLM, handle any tool calls that arise, and stream the response back. Each iteration updates the message history and conversational state.

Message Flow Architecture

The message flow follows a pipeline pattern. When a new user message arrives, it passes through several stages:

1. Preprocessing — The message is tokenized and analyzed for intent (question, command, follow-up, clarification response).
2. Context Assembly — The system prompt, relevant history, and any injected context (retrieved documents, tool results) are assembled into the prompt.
3. LLM Call — The assembled prompt is sent to the model with streaming enabled and tool definitions attached.
4. Response Handling — Streamed tokens are forwarded to the client. If a tool call is detected, execution branches to the tool handler.
5. History Update — The assistant’s response (and any tool results) are appended to the message history.

Context Window Sliding Strategy

A common implementation uses a sliding window with priority retention. The system prompt always occupies the first slot. Recent messages (the last N turns) are kept verbatim. Older messages are summarized into a single “conversation summary” block that captures key facts, user preferences, and prior decisions. Tool results older than a configurable threshold are replaced with their summary metadata (tool name, query, result count) rather than full payloads.

Implementation Patterns

Message History Store

The message store is the core data structure that accumulates conversation turns. Each message includes role, content, optional tool call metadata, and a token count estimate for budget management.

interface ChatMessage {
  role: "system" | "user" | "assistant" | "tool";
  content: string;
  toolCallId?: string;
  toolCalls?: ToolCall[];
  timestamp: number;
  tokenEstimate: number;
}

class MessageHistory {
  private messages: ChatMessage[] = [];
  private maxTokens: number;
  private systemPrompt: ChatMessage;

  constructor(systemPrompt: string, maxTokens: number = 120000) {
    this.systemPrompt = {
      role: "system",
      content: systemPrompt,
      timestamp: Date.now(),
      tokenEstimate: this.estimateTokens(systemPrompt),
    };
    this.maxTokens = maxTokens;
  }

  append(message: ChatMessage): void {
    this.messages.push(message);
    this.enforceTokenBudget();
  }

  private enforceTokenBudget(): void {
    let totalTokens = this.systemPrompt.tokenEstimate;
    // Count from newest to oldest
    const keep: ChatMessage[] = [];
    for (let i = this.messages.length - 1; i >= 0; i--) {
      const msg = this.messages[i];
      if (totalTokens + msg.tokenEstimate > this.maxTokens) {
        // Summarize remaining older messages
        const older = this.messages.slice(0, i + 1);
        const summary = this.summarize(older);
        keep.unshift(summary);
        break;
      }
      totalTokens += msg.tokenEstimate;
      keep.unshift(msg);
    }
    this.messages = keep;
  }

  getPromptMessages(): ChatMessage[] {
    return [this.systemPrompt, ...this.messages];
  }

  private estimateTokens(text: string): number {
    return Math.ceil(text.length / 4);
  }

  private summarize(messages: ChatMessage[]): ChatMessage {
    const facts = messages
      .filter((m) => m.role === "assistant" || m.role === "user")
      .map((m) => `${m.role}: ${m.content.slice(0, 200)}`)
      .join("\n");
    return {
      role: "system",
      content: `[Conversation summary - ${messages.length} earlier messages]\n${facts}`,
      timestamp: messages[0]?.timestamp ?? Date.now(),
      tokenEstimate: this.estimateTokens(facts),
    };
  }
}

Streaming Tool-Call Handler

When the LLM emits a tool call during streaming, the agent pauses text generation, executes the tool, and injects the result back as a tool message before resuming.

interface ToolCall {
  id: string;
  name: string;
  arguments: Record<string, unknown>;
}

interface StreamCallbacks {
  onToken: (token: string) => void;
  onToolStart: (toolName: string) => void;
  onToolEnd: (toolName: string, result: string) => void;
  onComplete: (fullResponse: string) => void;
}

async function handleStreamWithTools(
  history: MessageHistory,
  tools: Map<string, (args: Record<string, unknown>) => Promise<string>>,
  callbacks: StreamCallbacks,
): Promise<void> {
  const messages = history.getPromptMessages();

  const stream = await llm.chat({
    messages,
    tools: Array.from(tools.keys()).map((name) => ({
      type: "function",
      function: { name, parameters: toolSchemas.get(name) },
    })),
    stream: true,
  });

  let fullResponse = "";
  const pendingToolCalls: ToolCall[] = [];

  for await (const chunk of stream) {
    if (chunk.type === "content") {
      fullResponse += chunk.text;
      callbacks.onToken(chunk.text);
    } else if (chunk.type === "tool_call") {
      pendingToolCalls.push(chunk.toolCall);
    }
  }

  // Execute any tool calls
  for (const toolCall of pendingToolCalls) {
    callbacks.onToolStart(toolCall.name);
    const handler = tools.get(toolCall.name);
    if (!handler) throw new Error(`Unknown tool: ${toolCall.name}`);

    const result = await handler(toolCall.arguments);
    callbacks.onToolEnd(toolCall.name, result);

    // Inject tool result into history and recurse
    history.append({
      role: "assistant",
      content: fullResponse,
      toolCalls: pendingToolCalls,
      timestamp: Date.now(),
      tokenEstimate: Math.ceil(fullResponse.length / 4),
    });
    history.append({
      role: "tool",
      content: result,
      toolCallId: toolCall.id,
      timestamp: Date.now(),
      tokenEstimate: Math.ceil(result.length / 4),
    });

    // Recurse to let the model incorporate tool results
    return handleStreamWithTools(history, tools, callbacks);
  }

  // No tool calls — finalize
  history.append({
    role: "assistant",
    content: fullResponse,
    timestamp: Date.now(),
    tokenEstimate: Math.ceil(fullResponse.length / 4),
  });
  callbacks.onComplete(fullResponse);
}

Practical Applications

In a production chat application, the chat agent powers the conversational search interface. Users can ask natural language questions about their data, refine results through follow-up questions, and request actions like exporting or visualizing query results — all within a single continuous conversation.

Conversational Database Search

A user might say “Show me all orders from last week,” then follow up with “just the ones over $500,” then “export those as CSV.” Each message builds on the prior context. The agent must understand that “those” refers to the filtered result set, not all orders, and invoke the appropriate export tool with the correct query parameters.

Ambiguity Resolution in Practice

When a user asks “search for tables,” the agent must determine whether they mean database tables, HTML tables in documents, or furniture. The clarification system checks the conversation context — if prior messages discussed database schemas, it proceeds with database tables. If the context is ambiguous, it asks: “Do you mean database tables in your schema, or are you searching for something else?”

Tool Chaining Across Turns

Complex workflows span multiple turns with tool calls at each step. The agent might: (1) search the database to find matching records, (2) in the next turn, aggregate statistics on those records, and (3) in a third turn, generate a chart. Each tool call’s result informs the next turn’s context, and the message history preserves the full chain for coherent continuation.

Common Pitfalls

• Unbounded history growth — Without token budget enforcement, long conversations will exceed the context window and cause API errors. Always implement a sliding window or summarization strategy.
• Losing critical context during summarization — Aggressive summarization can drop facts the user expects the agent to remember (e.g., “I already told you my name”). Use priority-based retention for user-stated facts and preferences.
• Tool call loops — If the model repeatedly calls the same tool with the same arguments, the agent enters an infinite loop. Implement a maximum tool call depth per turn and detect repeated identical calls.
• Coreference resolution failures — Pronouns like “it,” “that,” and “those” require the agent to correctly identify their referent. When the referent is ambiguous, the agent should ask for clarification rather than guess wrong.
• Streaming partial tool calls — Tool call arguments arrive incrementally during streaming. Attempting to parse them before the full JSON is received causes errors. Buffer tool call chunks until the complete arguments are available.
• Stale context after tool failures — If a tool call fails (timeout, auth error), the agent must handle the error gracefully in context rather than proceeding as if the tool succeeded. Inject an error message into the history so the model can respond appropriately.