Building a useful knowledge base requires ingesting documentation from heterogeneous sources—Markdown files in Git repositories, HTML documentation sites, Google Docs, API references, and structured data files—and normalizing them into a single, searchable corpus. As an example, consider a documentation ingestion pipeline that handles source discovery, content extraction, automatic chunking, vector embedding, and indexing into a vector store.
The pipeline operates on the principle of source-agnostic normalization. Each source type (Markdown, HTML, Google Docs, etc.) has a dedicated adapter that extracts clean text and metadata, but the downstream stages—chunking, embedding, and indexing—operate on a uniform intermediate representation. This decoupling makes it straightforward to add new source types without modifying the core pipeline logic.
A critical challenge in documentation ingestion is maintaining coherence across chunks. A naive fixed-size splitter will break paragraphs mid-sentence and separate code examples from their explanations. The pipeline uses a structure-aware chunker that respects heading boundaries, code blocks, and paragraph breaks, producing chunks that are semantically self-contained while staying within embedding model token limits.
Each documentation source requires its own adapter to handle authentication, pagination, and format-specific extraction. The Markdown adapter reads files from the filesystem or clones Git repositories, parsing frontmatter for metadata. The Google Docs adapter uses the Google Docs API to retrieve document structure including headings, tables, and embedded images. The HTML adapter reuses the web crawler’s content extractor to strip boilerplate.
All source adapters produce the same intermediate format: a sequence of typed blocks (heading, paragraph, code, table, image-caption) with associated metadata (source URL, last modified timestamp, heading hierarchy). This uniform representation lets the chunker and embedder work identically regardless of the original source format.
The chunker splits documents along structural boundaries rather than at fixed character counts. It walks the block sequence and accumulates blocks into a chunk until adding the next block would exceed the token limit. When a split is needed, it prefers to break at heading boundaries, then paragraph boundaries, and only as a last resort mid-paragraph. Each chunk inherits the heading hierarchy from its parent document, so a chunk under “API Reference > Authentication > OAuth2” carries that full context even when retrieved in isolation.
When a chunk is retrieved via semantic search, the user needs to understand where it fits within the larger document. Prepending the heading hierarchy (e.g., “Guide > Setup > Docker”) to each chunk gives the embedding model and the end user structural context without duplicating the full document.
Each chunk is embedded using a vector model and stored in the vector store with its full metadata. The metadata includes the source URL, document title, heading path, last-modified timestamp, and a content hash for change detection. During re-ingestion, the pipeline compares content hashes to skip unchanged chunks, making incremental updates efficient.
Re-ingesting an entire documentation site on every update is wasteful. By storing a content hash (SHA-256 of the raw text) alongside each chunk, the pipeline can detect which chunks have actually changed and only re-embed and re-index those. This reduces embedding API costs and indexing load by orders of magnitude for large documentation corpora.
Beyond the raw text, the pipeline extracts and stores structured metadata: language detection results, code language annotations, link targets, and cross-references between documents. This metadata powers faceted search (e.g., “show only Go code examples”) and enables the retrieval system to boost results from authoritative sources.
The ingestion pipeline processes documentation sources through a series of stages, each feeding into the next via channels. The architecture is designed for concurrent execution and graceful error handling.
// IngestionPipeline orchestrates documentation ingestion.
type IngestionPipeline struct {
sources []SourceConfig
chunker *StructuredChunker
embedder Embedder
store VectorStore
hashIndex *ContentHashIndex
}
// SourceConfig describes a documentation source.
type SourceConfig struct {
Type string // "markdown", "html", "gdocs", "github"
URL string
Credentials string // Secret name for auth tokens.
Include []string // Glob patterns for files to include.
Exclude []string // Glob patterns for files to exclude.
}
// Ingest runs the full ingestion pipeline for all configured sources.
func (p *IngestionPipeline) Ingest(ctx context.Context) (*IngestReport, error) {
report := &IngestReport{}
for _, src := range p.sources {
adapter, err := NewAdapter(src)
if err != nil {
report.Errors = append(report.Errors, err)
continue
}
docs, err := adapter.Fetch(ctx)
if err != nil {
report.Errors = append(report.Errors, err)
continue
}
for _, doc := range docs {
blocks := adapter.Parse(doc)
chunks := p.chunker.Chunk(blocks)
for _, chunk := range chunks {
hash := sha256Hex(chunk.Text)
if p.hashIndex.Exists(chunk.SourceKey(), hash) {
report.Skipped++
continue
}
vec, err := p.embedder.Embed(ctx, chunk.Text)
if err != nil {
report.Errors = append(report.Errors, err)
continue
}
err = p.store.Upsert(ctx, &VectorRecord{
ID: chunk.ID(),
Vector: vec,
Metadata: chunk.Metadata(),
})
if err != nil {
report.Errors = append(report.Errors, err)
continue
}
p.hashIndex.Set(chunk.SourceKey(), hash)
report.Indexed++
}
}
}
return report, nil
}// StructuredChunker splits document blocks into token-limited chunks.
type StructuredChunker struct {
MaxTokens int
OverlapTokens int
Tokenizer Tokenizer
}
// Chunk splits a sequence of blocks into chunks respecting structure.
func (c *StructuredChunker) Chunk(blocks []Block) []Chunk {
var chunks []Chunk
var current []Block
currentTokens := 0
headingPath := []string{}
for _, block := range blocks {
blockTokens := c.Tokenizer.Count(block.Text)
// Update heading path on heading blocks.
if block.Type == BlockHeading {
headingPath = truncateToLevel(headingPath, block.Level)
headingPath = append(headingPath, block.Text)
}
// If adding this block exceeds the limit, flush.
if currentTokens+blockTokens > c.MaxTokens && len(current) > 0 {
chunks = append(chunks, Chunk{
Blocks: current,
HeadingPath: copySlice(headingPath),
})
current = nil
currentTokens = 0
}
current = append(current, block)
currentTokens += blockTokens
}
// Flush remaining blocks.
if len(current) > 0 {
chunks = append(chunks, Chunk{
Blocks: current,
HeadingPath: copySlice(headingPath),
})
}
return chunks
}A system like this can ingest documentation from dozens of sources—internal wikis, GitHub repositories, API documentation sites, and design documents—into a unified vector index. Engineers can then search across all documentation with a single semantic query, eliminating the need to remember which source contains which information.
By scheduling periodic re-ingestion runs, the pipeline keeps the knowledge base current. The content-hash-based change detection ensures that only truly updated content is re-processed, keeping the system efficient even when pointed at large, frequently-updated documentation sites.
The chunked and embedded documentation serves as the retrieval corpus for Retrieval-Augmented Generation (RAG) workflows. When a user asks a question, the system retrieves the most relevant chunks via vector similarity search, then feeds them as context to a language model. The structure-aware chunking ensures retrieved context is coherent and self-contained.
Source adapters are registered in a global registry keyed by source type string. The pipeline looks up the adapter factory at runtime, making it easy to add new source types via a single registration call. Each adapter implements the same interface: Fetch(ctx) ([]RawDoc, error) and Parse(RawDoc) []Block.
// AdapterFactory creates a source adapter from config.
type AdapterFactory func(SourceConfig) (Adapter, error)
var adapterRegistry = map[string]AdapterFactory{}
// RegisterAdapter registers a new source adapter type.
func RegisterAdapter(sourceType string, factory AdapterFactory) {
adapterRegistry[sourceType] = factory
}
// NewAdapter creates an adapter for the given source config.
func NewAdapter(cfg SourceConfig) (Adapter, error) {
factory, ok := adapterRegistry[cfg.Type]
if !ok {
return nil, fmt.Errorf("unknown source type: %s", cfg.Type)
}
return factory(cfg)
}
func init() {
RegisterAdapter("markdown", NewMarkdownAdapter)
RegisterAdapter("html", NewHTMLAdapter)
RegisterAdapter("gdocs", NewGDocsAdapter)
RegisterAdapter("github", NewGitHubAdapter)
}Embedding APIs have rate limits and per-request costs. The pipeline batches chunks into groups of 32–128 and sends them in a single API call, then waits for the rate limit window before the next batch. A semaphore controls concurrency, and exponential backoff handles transient failures.
// BatchEmbedder wraps an embedding API with batching and rate limiting.
type BatchEmbedder struct {
client EmbedClient
batchSize int
limiter *rate.Limiter
}
func (b *BatchEmbedder) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error) {
var allVecs [][]float32
for i := 0; i < len(texts); i += b.batchSize {
end := min(i+b.batchSize, len(texts))
batch := texts[i:end]
if err := b.limiter.Wait(ctx); err != nil {
return nil, err
}
vecs, err := b.client.Embed(ctx, batch)
if err != nil {
return nil, fmt.Errorf("embed batch %d-%d: %w", i, end, err)
}
allVecs = append(allVecs, vecs...)
}
return allVecs, nil
}authenticate() method with your API key” is useless without knowing it came from the OAuth2 section of the authentication guide. Always prepend the heading path to each chunk.