MCP Server for Persistent Agent Memory

Overview

The Model Context Protocol (MCP) defines a standardized interface through which AI agents — such as Claude, GPT-based assistants, and custom agentic systems — can interact with external tools and data sources. By building an MCP server dedicated to persistent memory, you give agents the ability to store, recall, and reason over information that persists across conversation boundaries. This transforms a stateless language model into a system that accumulates knowledge over time, remembering user preferences, prior decisions, project context, and domain-specific facts.

The core challenge with agent memory isn’t storage — it’s retrieval. When an agent has thousands of stored memories, finding the right ones at the right time is critical. Pure semantic search (embedding similarity) excels at finding conceptually related memories but can miss exact keyword matches. Pure full-text search (BM25) handles precise terminology well but fails on paraphrased or conceptually equivalent queries. Hybrid search combines both approaches and uses reciprocal rank fusion (RRF) to merge their ranked results into a single, high-quality result set.

In this topic, we build an MCP server in TypeScript that exposes memory storage and retrieval as MCP tools. The server persists memories in a vector database (such as Qdrant, Weaviate, or ChromaDB), generates embeddings for semantic search, maintains a full-text index for keyword matching, and fuses results using RRF. The result is a production-grade memory layer that any MCP-compatible agent can use without custom integration work.

Core Concepts

How It Works

MCP Server Setup

The MCP server is built using the official @modelcontextprotocol/sdk package. It registers tools for storing and retrieving memories, then listens over stdio transport so any MCP-compatible client can connect.

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import { MemoryStore } from "./memory-store.js";

const server = new McpServer({
  name: "agent-memory",
  version: "1.0.0",
});

const memoryStore = new MemoryStore({
  qdrantUrl: process.env.QDRANT_URL ?? "http://localhost:6333",
  collectionName: "agent_memories",
  embeddingModel: "text-embedding-3-small",
  embeddingDimensions: 1536,
});

// Tool: store a new memory
server.tool(
  "store_memory",
  "Store a piece of information for later retrieval",
  {
    content: z.string().describe("The text content to remember"),
    tags: z.array(z.string()).optional().describe("Categorization tags"),
    importance: z.enum(["low", "medium", "high"]).default("medium"),
    source: z.string().optional().describe("Where this memory came from"),
  },
  async ({ content, tags, importance, source }) => {
    const id = await memoryStore.store({ content, tags, importance, source });
    return { content: [{ type: "text", text: `Stored memory ${id}` }] };
  }
);

// Tool: search memories using hybrid retrieval
server.tool(
  "search_memories",
  "Search stored memories using semantic and keyword matching",
  {
    query: z.string().describe("The search query"),
    limit: z.number().default(10).describe("Max results to return"),
    tags: z.array(z.string()).optional().describe("Filter by tags"),
    minImportance: z.enum(["low", "medium", "high"]).optional(),
  },
  async ({ query, limit, tags, minImportance }) => {
    const results = await memoryStore.hybridSearch(query, {
      limit, tags, minImportance,
    });
    const formatted = results.map((r, i) =>
      `[${i + 1}] (score: ${r.score.toFixed(3)}) ${r.content}`
    ).join("\n\n");
    return { content: [{ type: "text", text: formatted || "No memories found." }] };
  }
);

// Tool: delete a memory
server.tool(
  "delete_memory",
  "Remove a stored memory by its ID",
  { id: z.string().describe("The memory ID to delete") },
  async ({ id }) => {
    await memoryStore.delete(id);
    return { content: [{ type: "text", text: `Deleted memory ${id}` }] };
  }
);

const transport = new StdioServerTransport();
await server.connect(transport);

Embedding Generation

Each memory is embedded at storage time. The embedding is generated via the OpenAI embeddings API (or any compatible provider). We cache embeddings alongside the text to avoid redundant API calls on re-indexing.

import OpenAI from "openai";

export class EmbeddingService {
  private client: OpenAI;
  private model: string;

  constructor(model = "text-embedding-3-small") {
    this.client = new OpenAI();
    this.model = model;
  }

  async embed(texts: string[]): Promise<number[][]> {
    const response = await this.client.embeddings.create({
      model: this.model,
      input: texts,
    });
    return response.data.map((d) => d.embedding);
  }

  async embedSingle(text: string): Promise<number[]> {
    const [embedding] = await this.embed([text]);
    return embedding;
  }
}

Hybrid Search with Reciprocal Rank Fusion

The hybrid search implementation runs semantic and full-text searches in parallel, then fuses their ranked results using RRF. This ensures that memories matching both semantically and by keywords rank highest, while memories found by only one method still appear in results.

interface SearchResult {
  id: string;
  content: string;
  score: number;
  metadata: Record<string, unknown>;
}

interface SearchOptions {
  limit: number;
  tags?: string[];
  minImportance?: string;
}

export class MemoryStore {
  private qdrant: QdrantClient;
  private embedder: EmbeddingService;
  private collection: string;

  // ... constructor omitted for brevity ...

  async hybridSearch(
    query: string,
    opts: SearchOptions
  ): Promise<SearchResult[]> {
    const queryEmbedding = await this.embedder.embedSingle(query);
    const filter = this.buildFilter(opts.tags, opts.minImportance);

    // Run both searches in parallel
    const [semanticResults, fullTextResults] = await Promise.all([
      this.semanticSearch(queryEmbedding, filter, opts.limit * 2),
      this.fullTextSearch(query, filter, opts.limit * 2),
    ]);

    // Fuse results with RRF
    const fused = this.reciprocalRankFusion(
      [semanticResults, fullTextResults],
      { k: 60 }
    );

    return fused.slice(0, opts.limit);
  }

  private reciprocalRankFusion(
    rankedLists: SearchResult[][],
    options: { k: number }
  ): SearchResult[] {
    const scoreMap = new Map<string, { score: number; result: SearchResult }>();

    for (const rankedList of rankedLists) {
      for (let rank = 0; rank < rankedList.length; rank++) {
        const result = rankedList[rank];
        const rrfScore = 1 / (options.k + rank + 1);

        if (scoreMap.has(result.id)) {
          scoreMap.get(result.id)!.score += rrfScore;
        } else {
          scoreMap.set(result.id, { score: rrfScore, result });
        }
      }
    }

    return Array.from(scoreMap.values())
      .sort((a, b) => b.score - a.score)
      .map(({ score, result }) => ({ ...result, score }));
  }

  private async semanticSearch(
    embedding: number[],
    filter: object | undefined,
    limit: number
  ): Promise<SearchResult[]> {
    const results = await this.qdrant.search(this.collection, {
      vector: embedding,
      limit,
      filter,
      with_payload: true,
    });

    return results.map((r) => ({
      id: String(r.id),
      content: r.payload?.content as string,
      score: r.score,
      metadata: r.payload as Record<string, unknown>,
    }));
  }

  private async fullTextSearch(
    query: string,
    filter: object | undefined,
    limit: number
  ): Promise<SearchResult[]> {
    // Qdrant full-text search via scroll with match condition
    const results = await this.qdrant.scroll(this.collection, {
      filter: {
        must: [
          { key: "content", match: { text: query } },
          ...(filter ? [filter] : []),
        ],
      },
      limit,
      with_payload: true,
      with_vector: false,
    });

    return results.points.map((r, index) => ({
      id: String(r.id),
      content: r.payload?.content as string,
      score: 1 / (index + 1), // Proxy score based on position
      metadata: r.payload as Record<string, unknown>,
    }));
  }

  private buildFilter(tags?: string[], minImportance?: string) {
    const conditions: object[] = [];
    if (tags && tags.length > 0) {
      conditions.push({
        key: "tags",
        match: { any: tags },
      });
    }
    if (minImportance) {
      const levels = ["low", "medium", "high"];
      const minIdx = levels.indexOf(minImportance);
      const allowed = levels.slice(minIdx);
      conditions.push({
        key: "importance",
        match: { any: allowed },
      });
    }
    return conditions.length > 0 ? { must: conditions } : undefined;
  }
}

Memory Storage and Lifecycle

When storing a memory, the system generates an embedding, assigns metadata (timestamp, tags, importance, a unique ID), and upserts the point into the vector database. Memory lifecycle management includes TTL-based expiration, importance-based retention, and deduplication via similarity thresholds.

import { v4 as uuidv4 } from "uuid";

interface MemoryInput {
  content: string;
  tags?: string[];
  importance?: string;
  source?: string;
}

// Inside MemoryStore class:

async store(input: MemoryInput): Promise<string> {
  const id = uuidv4();
  const embedding = await this.embedder.embedSingle(input.content);

  // Deduplication: check if a very similar memory already exists
  const similar = await this.qdrant.search(this.collection, {
    vector: embedding,
    limit: 1,
    score_threshold: 0.95,
  });

  if (similar.length > 0) {
    // Update existing memory instead of creating duplicate
    const existingId = similar[0].id;
    await this.qdrant.upsert(this.collection, {
      points: [{
        id: existingId,
        vector: embedding,
        payload: {
          content: input.content,
          tags: input.tags ?? [],
          importance: input.importance ?? "medium",
          source: input.source ?? "agent",
          createdAt: similar[0].payload?.createdAt,
          updatedAt: new Date().toISOString(),
          accessCount: ((similar[0].payload?.accessCount as number) ?? 0) + 1,
        },
      }],
    });
    return String(existingId);
  }

  // Store new memory
  await this.qdrant.upsert(this.collection, {
    points: [{
      id,
      vector: embedding,
      payload: {
        content: input.content,
        tags: input.tags ?? [],
        importance: input.importance ?? "medium",
        source: input.source ?? "agent",
        createdAt: new Date().toISOString(),
        updatedAt: new Date().toISOString(),
        accessCount: 0,
      },
    }],
  });

  return id;
}

async delete(id: string): Promise<void> {
  await this.qdrant.delete(this.collection, { points: [id] });
}

async expireOldMemories(maxAgeDays: number): Promise<number> {
  const cutoff = new Date();
  cutoff.setDate(cutoff.getDate() - maxAgeDays);

  const old = await this.qdrant.scroll(this.collection, {
    filter: {
      must: [
        { key: "importance", match: { value: "low" } },
        { key: "updatedAt", range: { lt: cutoff.toISOString() } },
      ],
    },
    limit: 1000,
  });

  const ids = old.points.map((p) => p.id);
  if (ids.length > 0) {
    await this.qdrant.delete(this.collection, { points: ids });
  }
  return ids.length;
}

Implementation Patterns

Collection Initialization with Indexes

Before the MCP server can serve requests, it must ensure the Qdrant collection exists with the correct vector configuration and a full-text index on the content field. This initialization runs once at startup.

import { QdrantClient } from "@qdrant/js-client-rest";

async function initializeCollection(
  client: QdrantClient,
  name: string,
  dimensions: number
): Promise<void> {
  const collections = await client.getCollections();
  const exists = collections.collections.some((c) => c.name === name);

  if (!exists) {
    await client.createCollection(name, {
      vectors: {
        size: dimensions,
        distance: "Cosine",
      },
      optimizers_config: {
        default_segment_number: 2,
      },
    });

    // Create full-text index for BM25-style keyword search
    await client.createPayloadIndex(name, {
      field_name: "content",
      field_schema: {
        type: "text",
        tokenizer: "word",
        min_token_len: 2,
        max_token_len: 20,
        lowercase: true,
      },
    });

    // Create keyword indexes for filtered queries
    await client.createPayloadIndex(name, {
      field_name: "tags",
      field_schema: "keyword",
    });

    await client.createPayloadIndex(name, {
      field_name: "importance",
      field_schema: "keyword",
    });

    console.error(`Created collection "${name}" with full-text index`);
  }
}

MCP Client Configuration

To connect an MCP client (e.g., Claude Desktop) to this memory server, add the following to your MCP client configuration file. The server runs as a subprocess communicating over stdio.

{
  "mcpServers": {
    "agent-memory": {
      "command": "node",
      "args": ["dist/server.js"],
      "env": {
        "QDRANT_URL": "http://localhost:6333",
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

Contextual Memory Retrieval with Metadata Enrichment

In practice, raw search results need enrichment — adding recency weighting, access-count boosting, and context windows. This pattern adjusts RRF scores based on temporal relevance and usage patterns.

function applyTemporalBoost(
  results: SearchResult[],
  halfLifeDays: number = 30
): SearchResult[] {
  const now = Date.now();

  return results.map((r) => {
    const updatedAt = new Date(r.metadata.updatedAt as string).getTime();
    const ageDays = (now - updatedAt) / (1000 * 60 * 60 * 24);

    // Exponential decay: score *= 2^(-age/halfLife)
    const decayFactor = Math.pow(2, -ageDays / halfLifeDays);

    // Access frequency boost (log scale to avoid runaway)
    const accessCount = (r.metadata.accessCount as number) ?? 0;
    const accessBoost = 1 + Math.log2(1 + accessCount) * 0.1;

    // Importance multiplier
    const importanceMultipliers: Record<string, number> = {
      high: 1.3,
      medium: 1.0,
      low: 0.7,
    };
    const importanceFactor =
      importanceMultipliers[r.metadata.importance as string] ?? 1.0;

    return {
      ...r,
      score: r.score * decayFactor * accessBoost * importanceFactor,
    };
  }).sort((a, b) => b.score - a.score);
}

Common Pitfalls

• Unbounded memory growth: Without a retention policy, the memory store grows indefinitely. Implement TTL-based expiration for low-importance memories and periodic deduplication passes to keep the collection manageable.
• Embedding model mismatch: If you change the embedding model after memories have been stored, existing vectors become incompatible with new query vectors. Always re-embed the entire collection when switching models, or maintain separate collections per model version.
• Over-relying on semantic search alone: Embeddings compress meaning into fixed-dimension vectors and inevitably lose information. Exact identifiers, error codes, UUIDs, and domain-specific abbreviations are poorly served by embeddings alone — full-text search is essential for these cases.
• RRF k-parameter tuning: The constant k=60 is a widely-used default, but it controls how much rank position matters. Smaller k values amplify differences between top-ranked results; larger values flatten the distribution. Benchmark with your actual query patterns before changing it.
• Storing raw conversation turns as memories: Dumping entire conversation messages into memory creates noise. Extract and summarize the key facts, decisions, or preferences before storing. A memory like “User prefers PostgreSQL over MySQL for new projects” is far more retrievable than a raw 500-word exchange.
• Missing metadata filters: Without payload indexes on fields like tags and importance, filtered queries fall back to brute-force scanning. Always create payload indexes for fields you plan to filter on.
• Ignoring embedding latency: Embedding API calls add 50–200ms per memory storage or retrieval. Batch embeddings where possible and consider caching query embeddings for repeated or similar queries within a session.