A content-addressed agent memory store keeps durable artifacts outside the model context and identifies stored bytes by cryptographic hash rather than by mutable filenames or database row locations. The pattern is useful for assistants that repeatedly ingest code, documents, transcripts, research files, and generated artifacts across long-running work. Instead of duplicating the same bytes every time a file reappears, the store splits content into chunks, hashes each chunk, deduplicates repeats, and builds searchable metadata on top.
The YAMS project is a concrete reference architecture for this pattern. Its README describes SHA-256 content-addressed storage with block-level deduplication, Rabin chunking, compression, full-text search through SQLite FTS5, vector search, Tree-sitter symbol extraction, Merkle tree snapshot diffs, write-ahead-log durability, a CLI, and an MCP server. The reusable lesson is not the specific command surface; it is the layering: immutable content store first, searchable metadata second, agent-facing tools last.
This architecture gives agent systems a memory substrate that can be repaired, audited, and queried without stuffing raw archives into every prompt. Agents can retrieve exact files by hash, search extracted text, find semantically related artifacts, compare snapshots, or call an MCP tool for bounded results. Engineers can rebuild indexes and embeddings because the underlying content remains stable and addressable.
A content address is a digest computed from the content itself. If two chunks have the same hash, they are treated as the same stored object. That makes deduplication natural: repeated source files, copied document fragments, generated build outputs, and recurring attachments can share physical storage while keeping separate metadata records that explain where each object appeared.
Fixed-size chunks are cheap, but a one-byte insertion near the front of a file shifts every following boundary and ruins deduplication. Content-defined chunking uses a rolling fingerprint, such as Rabin fingerprinting, to choose chunk boundaries from the byte stream itself. Local edits tend to disturb only nearby chunks, so large unchanged regions retain their hashes across revisions.
Agents rarely want raw bytes alone. They need extracted text, MIME type, language, symbols, tags, timestamps, vector embeddings, and ranking signals. A practical memory store separates immutable chunks from mutable indexes so full-text search, semantic retrieval, and symbol graphs can be rebuilt when extractors or embedding models improve.
The stored content should be stable and hash-addressed, while the derived understanding can evolve. You can re-extract text, regenerate embeddings, rebuild FTS5 tables, or add graph edges without changing the identity of the source chunks.
The ingestion path reads a file or document, splits it into content-defined chunks, hashes each chunk, writes missing chunks to the store, and records a manifest that maps the logical document to chunk hashes. The logical document record can change name, tags, or ownership without rewriting chunks that are already present.
def ingest_document(path, bytes_in, metadata):
chunk_refs = []
for chunk in rabin_chunks(bytes_in, min_size=4096, avg_size=32768, max_size=262144):
digest = sha256(chunk)
if not chunk_store.exists(digest):
packed = zstd_compress(chunk)
chunk_store.put(digest, packed)
chunk_refs.append({"sha256": digest, "length": len(chunk)})
doc_hash = sha256_json(chunk_refs)
document_table.upsert({
"doc_hash": doc_hash,
"path": path,
"chunks": chunk_refs,
"metadata": metadata,
})
return doc_hashYAMS documents several extraction paths that converge on a shared extraction utility: normal post-ingest indexing, background repair, manual repair commands, diagnostic repair, and single-document reindexing. The important design choice is convergence. Whether work starts from a daemon event, a CLI command, or a user interface action, it should call the same extraction and indexing machinery.
def index_document(doc_hash):
doc = document_table.get(doc_hash)
raw = chunk_store.reassemble(doc.chunks)
text = extract_text(raw, mime=doc.metadata["mime"], extension=doc.metadata["extension"])
if not text:
return {"indexed": False, "reason": "no extractor"}
content_table.upsert(doc_hash=doc_hash, text=text)
fts5_index.replace(doc_hash, text)
embedding_queue.enqueue(doc_hash, text)
symbol_queue.enqueue_if_source_code(doc_hash, raw, doc.metadata)The agent should not receive direct access to the storage engine. It should receive narrow operations: add content, search content, retrieve by hash, inspect metadata, compare snapshots, or fetch a bounded excerpt. MCP is a natural interface for this because it can describe tools and return structured results while keeping the bulk store outside the model context.
{
"name": "memory_search",
"description": "Search stored artifacts using keyword and semantic retrieval.",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string" },
"max_results": { "type": "integer", "minimum": 1, "maximum": 20 },
"path_prefix": { "type": ["string", "null"] }
},
"required": ["query"]
}
}A coding agent can ingest repositories, diffs, build logs, generated artifacts, and design documents into the same memory store. Content hashes make repeated checkouts cheap, Tree-sitter extraction can populate symbol-aware search, and snapshot diffs help the agent reason about what changed between two runs.
Research agents often reprocess the same PDFs, HTML pages, datasets, and extracted text through multiple experiments. A content-addressed store lets the system cache raw artifacts once, then rebuild text indexes, embeddings, and derived summaries as retrieval strategies change.
When an assistant claims that a fact came from a stored artifact, the application can attach a content hash, offset, excerpt, and metadata trail. That is stronger than a free-floating model memory because engineers can reconstruct the source bytes and rerun the extraction path that produced the evidence.
The object store should optimize immutable chunk storage: hash lookup, compression, packing, reference counts, garbage collection, and integrity checks. The metadata store should optimize queries: document records, tags, path history, access scope, FTS5 rows, embedding dimensions, symbol tables, and repair status.
CREATE TABLE documents (
doc_hash TEXT PRIMARY KEY,
logical_path TEXT NOT NULL,
mime_type TEXT,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
chunk_manifest_json TEXT NOT NULL,
extraction_status TEXT NOT NULL DEFAULT 'pending',
embedding_status TEXT NOT NULL DEFAULT 'pending'
);
CREATE VIRTUAL TABLE document_fts
USING fts5(doc_hash UNINDEXED, title, body, tokenize='porter unicode61');
CREATE TABLE vector_index_metadata (
doc_hash TEXT PRIMARY KEY,
embedding_model TEXT NOT NULL,
embedding_dim INTEGER NOT NULL,
generated_at INTEGER NOT NULL
);Indexes drift. Extractors fail. Embedding dimensions change. Optional plugins are installed after content was ingested. Treat repair as a normal maintenance path rather than an emergency script. A repair service should scan for missing text, stale embeddings, wrong dimensions, and broken graph entries, then rebuild only affected derivatives from the hash-addressed source.
A Merkle snapshot can represent a directory, workspace, or corpus state as a tree of hashes. Comparing two roots identifies changed subtrees without byte-by-byte scans over the whole workspace. For agent systems, that enables fast “what changed since the last run?” queries and safe provenance for generated context.