A CRM Markdown Data Lake is an architecture pattern where client records are stored as individual markdown files with structured YAML or JSON frontmatter, rather than in a traditional relational database. Each file represents a single entity—a company, contact, or deal—with machine-readable metadata in the frontmatter and human-readable narrative in the body. This approach turns a Git repository into a queryable, version-controlled CRM that both humans and AI agents can read and write naturally.
The key challenge with thousands of markdown files is discovery: an agent asked “which enterprise clients in fintech haven’t been contacted in 90 days?” cannot afford to parse every file on every query. The solution is a machine-generated summary index—typically a summary.json file that catalogs every record’s frontmatter fields into a single queryable structure. This index is regenerated whenever records change, giving agents a fast lookup table while the markdown files remain the source of truth.
The agent platform’s CRM data lake implements this pattern at scale, maintaining thousands of client records as markdown files with rich frontmatter (tiers, tags, tech stacks, last-contact dates) and generating summary catalogs that power agent queries, pipeline filtering, and dashboard views without requiring a database server.
Each markdown file begins with a YAML or JSON frontmatter block delimited by --- markers. This block serves as the record’s structured schema: fields like company_name, tier, tags, tech_stack, last_contact, and status are machine-parseable while the markdown body below contains free-form notes, meeting summaries, and context that humans and LLMs read.
The frontmatter/body split mirrors the structured/unstructured divide in traditional databases. Frontmatter fields are indexable, filterable, and sortable—like SQL columns. The markdown body is full-text searchable and LLM-interpretable—like a document store. A single file gives you both capabilities without schema migrations or ORM mappings.
The summary index is a denormalized JSON file that aggregates frontmatter from all records into a single array. Each entry contains the file path, all frontmatter fields, and optionally computed fields like “days since last contact” or “engagement score.” The index is regenerated by a build script that walks the markdown directory, parses each file’s frontmatter, and writes the consolidated JSON. Because it’s derived data, it can be safely regenerated at any time.
The summary.json is never edited directly. It is always regenerated from the markdown files. This means the markdown files are the single source of truth, and the index is a read-optimized projection. If the index becomes corrupted or out of date, deleting and regenerating it is always safe. This pattern is analogous to a materialized view in a database.
By storing records as files in a Git repository, you get version history, branching, merge conflict resolution, and audit trails for free. Every edit to a client record is a commit with a timestamp, author, and diff. Rolling back a bad edit is git revert. Reviewing what changed in a quarter is git log --since. Multiple agents or humans can work on different records in parallel and merge without coordination.
A common access pattern is finding records sorted by recency—“show me clients contacted in the last 30 days” or “which leads went cold?” The summary index stores last_contact as an ISO 8601 date, enabling simple date arithmetic in any language. Sorting the index by this field gives an instant recency view without scanning individual files.
Records are classified using two orthogonal dimensions: tiers (priority levels like enterprise, mid-market, startup) and tags (descriptive labels like fintech, healthcare, active-deal). Tags are stored as arrays in frontmatter, enabling multi-label classification. The summary index supports filtering by any combination of tiers and tags, giving agents flexible query capabilities.
The data lake pipeline has three phases: record creation/update, index generation, and query serving. Records are created by humans editing markdown or by automated pipelines that enrich records from external data sources. The index generator runs as a build step, and queries are served by loading the index into memory and filtering.
summary.json.summary.json and filter/sort by any combination of fields.// TypeScript: Parse markdown frontmatter and build summary index
import * as fs from "fs";
import * as path from "path";
import matter from "gray-matter";
interface ClientRecord {
file: string;
company_name: string;
tier: string;
tags: string[];
tech_stack: string[];
last_contact: string;
status: string;
days_since_contact?: number;
}
function buildSummaryIndex(dataDir: string): ClientRecord[] {
const records: ClientRecord[] = [];
const files = fs.readdirSync(dataDir).filter(f => f.endsWith(".md"));
const now = new Date();
for (const file of files) {
const content = fs.readFileSync(path.join(dataDir, file), "utf-8");
const { data } = matter(content);
const lastContact = new Date(data.last_contact);
const daysSince = Math.floor(
(now.getTime() - lastContact.getTime()) / (1000 * 60 * 60 * 24)
);
records.push({
file,
company_name: data.company_name ?? path.basename(file, ".md"),
tier: data.tier ?? "unknown",
tags: data.tags ?? [],
tech_stack: data.tech_stack ?? [],
last_contact: data.last_contact ?? "",
status: data.status ?? "active",
days_since_contact: daysSince,
});
}
// Sort by recency (most recently contacted first)
records.sort((a, b) =>
(a.days_since_contact ?? Infinity) - (b.days_since_contact ?? Infinity)
);
return records;
}
// Generate and write the summary index
const index = buildSummaryIndex("./clients");
fs.writeFileSync("summary.json", JSON.stringify(index, null, 2));// TypeScript: Query the summary index for agent use
interface QueryFilter {
tier?: string;
tags?: string[];
maxDaysSinceContact?: number;
status?: string;
}
function queryClients(
index: ClientRecord[],
filter: QueryFilter
): ClientRecord[] {
return index.filter(record => {
if (filter.tier && record.tier !== filter.tier) return false;
if (filter.tags && !filter.tags.every(t => record.tags.includes(t))) {
return false;
}
if (
filter.maxDaysSinceContact !== undefined &&
(record.days_since_contact ?? Infinity) > filter.maxDaysSinceContact
) {
return false;
}
if (filter.status && record.status !== filter.status) return false;
return true;
});
}
// Example: Find enterprise fintech clients contacted in last 30 days
const results = queryClients(index, {
tier: "enterprise",
tags: ["fintech"],
maxDaysSinceContact: 30,
});
console.log(`Found ${results.length} matching clients`);In the agent platform’s CRM data lake, AI agents receive natural language queries like “find all healthcare clients we haven’t contacted in 60 days.” The agent loads summary.json, filters by the healthcare tag and days_since_contact > 60, and returns the matching records—all without a database connection. The agent can then read individual markdown files for detailed context before drafting outreach.
Marketing and sales pipelines use the summary index to segment clients for campaigns. A script might select all mid-market clients with the active-deal tag and generate personalized outreach templates. The segmentation logic is a simple JSON filter, making it easy to compose complex queries without SQL.
The summary index powers lightweight dashboards that show pipeline health: how many clients per tier, average days since last contact, tag distribution, and engagement trends over time. Because the index is a static JSON file, dashboards can be served from a CDN without a backend.
Markdown with frontmatter is arguably the most LLM-friendly data format. The frontmatter gives structure for precise filtering, while the markdown body gives context for reasoning. An LLM agent can read a client file and immediately understand the relationship history, technical context, and current status—something that requires joining multiple tables in a traditional CRM database.
Each entity gets exactly one markdown file. The filename is a slug derived from the entity name (e.g., acme-corp.md). This ensures a 1:1 mapping between files and records, making it impossible to have duplicate or orphaned entries. Directory structure can mirror organizational hierarchy: clients/enterprise/acme-corp.md.
For large data lakes (10,000+ records), full index regeneration on every change becomes slow. An incremental approach watches for file changes (via git diff or filesystem events), parses only changed files, and patches the existing index. The full regeneration serves as a consistency check that runs on a schedule.
// Incremental index update using git diff
import { execSync } from "child_process";
function getChangedFiles(since: string): string[] {
const diff = execSync(
`git diff --name-only ${since} -- clients/`
).toString();
return diff.trim().split("\n").filter(Boolean);
}
function patchIndex(
index: ClientRecord[],
changedFiles: string[]
): ClientRecord[] {
const updated = index.filter(
r => !changedFiles.includes(r.file)
);
for (const file of changedFiles) {
if (fs.existsSync(file)) {
const content = fs.readFileSync(file, "utf-8");
const { data } = matter(content);
updated.push(parseRecord(file, data));
}
// If file was deleted, it's already filtered out
}
return updated.sort(
(a, b) => (a.days_since_contact ?? Infinity)
- (b.days_since_contact ?? Infinity)
);
}A pre-commit hook or CI check validates that every markdown file’s frontmatter conforms to the expected schema: required fields are present, tiers come from an allowed set, tags are normalized to lowercase, and dates are valid ISO 8601. This catches data quality issues before they pollute the index.
The data lake works because of strict conventions: one file per entity, predictable frontmatter fields, slug-based filenames, and a well-known index location. There is no configuration file mapping fields to columns—the conventions are the schema. This makes the system easy to understand but requires discipline to maintain.
last_contact vs lastContact vs contacted_at). Without validation, the index silently drops records with non-standard fields. Solution: enforce a schema with a linter.summary.json will conflict because both branches wrote the full file. Solution: add summary.json to .gitattributes with merge=ours and regenerate after merge, or simply .gitignore the index and generate it at build time.summary.json into memory on every query is wasteful. For very large data lakes, consider SQLite as an intermediate index—still file-based and serverless, but with proper indexing and query optimization.fin-tech, fintech, financial-technology) that fragment queries. Maintain an allowed tags list and normalize on write.