Most developer documentation is optimized for browsers: navigation trees, search widgets, code tabs, client-side rendering, examples split across pages, and marketing wrappers around technical content. That presentation is useful for humans, but it is noisy input for coding agents and retrieval systems. An LLM-friendly documentation export gives agents a clean, text-first entry point into the same knowledge without forcing them to scrape and simplify every HTML page at inference time.
The emerging /llms.txt pattern treats documentation context as a first-class artifact. A site publishes a small Markdown index at a predictable path, usually /llms.txt, with a project summary and curated links to the pages an assistant should read. Many documentation sites also publish /llms-full.txt, a larger concatenated corpus that contains the full documentation in one machine-readable file. Astro’s documentation is a concrete example: its llms-full.txt file exposes framework concepts, guides, references, and examples as plain text rather than as a rendered web app.
This is a context-engineering technique, not a crawler control mechanism. It does not replace robots.txt, authentication, API reference design, or retrieval evaluation. Its value is operational: give agents and IDE assistants a stable, low-friction source of authoritative context, then make the size and structure predictable enough for indexing, chunking, cache warming, and on-demand retrieval.
An llms.txt file is a concise Markdown briefing. The proposed format starts with an H1 title, optionally includes a blockquote summary, then lists important resources under H2 section headings. Each list item points to a source page, preferably a clean Markdown version, and may include a short description. The file is small enough for an agent to read before deciding which deeper resources to fetch.
An llms-full.txt file is the opposite tradeoff: instead of only mapping the corpus, it bundles the corpus. This helps when the documentation set is small enough for a long-context model, when an IDE wants to index a single URL, or when a retrieval service wants a deterministic snapshot. It can be too large for direct prompting, so production systems usually chunk, embed, cache, or selectively summarize it.
Clean Markdown mirrors reduce parsing ambiguity. Instead of asking an agent to infer which parts of a client-rendered page are documentation, a site can expose the same content as Markdown with code blocks, headings, lists, and links preserved. This improves downstream chunk boundaries and makes examples easier to quote, test, and attribute.
LLM-friendly exports turn documentation from a human-only website into an addressable context layer. The index tells an agent what exists, the full export gives bulk access, and Markdown mirrors preserve page-level granularity for targeted retrieval.
The small index should describe the project and link to the highest-value pages. Do not blindly dump every URL from a sitemap. The purpose is to help an agent quickly choose relevant context, so the descriptions should explain what each linked page contains and when it matters.
# ExampleDB
> ExampleDB is an embedded search database with BM25, vector search, and transactional metadata.
Use the architecture guide before answering storage or indexing questions.
Use the API reference only when a concrete method signature is required.
## Core Docs
- [Quick start](https://docs.example.com/quickstart.md): Build a local index and run a first query.
- [Architecture](https://docs.example.com/architecture.md): Storage layout, segment lifecycle, query planner, and compaction.
- [Query API](https://docs.example.com/api/query.md): Request schema, filters, ranking options, and response fields.
## Optional
- [Changelog](https://docs.example.com/changelog.md): Release notes that can usually be skipped for general questions.A documentation build can emit Markdown beside HTML. The generator should preserve headings, code fences, tables, admonitions, and canonical URLs while removing navigation chrome and repeated footers. If the docs are framework-driven, generate mirrors from the same source files used to render the site so the two outputs do not drift.
# Conceptual build step for a docs site.
for page in docs_pages:
html = render_html(page)
markdown = render_markdown(page)
write(public_path(page), html)
write(public_path(page) + ".md", markdown)
write("public/llms.txt", build_llms_index(docs_pages))
write("public/llms-full.txt", concatenate_markdown(docs_pages))An agent should not always stuff llms-full.txt into the prompt. A better workflow reads llms.txt, selects a small set of pages, fetches the Markdown mirrors, and only falls back to the full export when broad coverage is needed. Retrieval systems can preprocess the full export into chunks with source URLs retained for citation.
def choose_context(question, llms_index):
candidates = parse_markdown_links(llms_index)
ranked = rank_by_title_and_description(question, candidates)
return ranked[:5]
index = fetch("https://docs.example.com/llms.txt")
for doc in choose_context(user_question, index):
context.append(fetch(doc.url))IDE assistants and coding agents often fail because their training data lags behind current library APIs. A project-level llms.txt lets a developer attach the current docs to a session without manually searching for the right guide. The agent can read the current route conventions, component APIs, configuration syntax, or migration notes before editing code.
For documentation chatbots, a full-text export becomes a reliable seed corpus. Instead of crawling rendered pages and stripping boilerplate, the indexing job can fetch a stable text endpoint, split it by headings or source-page boundaries, attach canonical URLs, and build embeddings. The resulting index is easier to reproduce and evaluate because the input artifact is deterministic.
Documentation tools can expose llms.txt as a discovery step. A tool can fetch the index, present available sections, retrieve page-level Markdown, or search the full export. This turns a generic web-fetch tool into a documentation-aware context provider with better source boundaries and less irrelevant page chrome.
The best exports are produced by the documentation pipeline. If the Markdown source, generated API reference, and route manifest already exist at build time, use those inputs directly. Scraping the final site reintroduces the exact ambiguity the export is meant to remove.
{
"scripts": {
"build:docs": "astro build",
"build:llms": "node scripts/build-llms.js",
"build": "npm run build:docs && npm run build:llms"
}
}A full export should not become an undifferentiated blob. Include page titles, canonical URLs, and clear separators so downstream chunkers can retain provenance. This matters for citations, freshness checks, and targeted reindexing when only one page changes.
# Architecture
Source: https://docs.example.com/architecture
...page content...
---
# Query API
Source: https://docs.example.com/api/query
...page content...Large reference material, generated API listings, changelogs, and exhaustive examples can overwhelm the full export. Put secondary material under an Optional section in the index, publish separate module-specific exports, or generate both compact and full variants. The goal is to make the context budget predictable rather than merely large.
llms.txt is not a permission system. Use authentication, authorization, and crawler controls for policy enforcement.