Building a knowledge base from web content requires more than downloading HTML. A full website crawling and content extraction pipeline handles robots.txt compliance, redirect chains, content deduplication, structured content extraction from noisy HTML, and indexing the results into a vector store. This is the system that turns the open web into a structured, searchable corpus.
The crawler operates in three phases. First, the discovery phase identifies pages to fetch—seeded from sitemaps, RSS feeds, or explicit URL lists. Second, the fetch phase retrieves each page, following redirects, respecting rate limits, and caching raw responses. Third, the extraction phase strips navigation, ads, and boilerplate from the HTML to produce clean content that is chunked, embedded, and indexed. Each phase is decoupled via an internal queue, allowing them to run at different rates and be scaled independently.
The deduplication system deserves special attention. The web is full of duplicate content: printer-friendly pages, paginated archives, syndicated articles, and URL variants that all serve the same text. The crawler uses content-based fingerprinting (SimHash) to detect near-duplicate pages and retain only the canonical version, dramatically reducing index bloat and improving search quality.
The robots.txt file is fetched and parsed before any URLs on a domain are visited. The crawler caches parsed rulesets per domain and checks every URL against the rules before enqueuing it. The implementation follows the Google robots.txt specification, handling wildcards, $ end-of-URL anchors, and multiple user-agent blocks. Crawl-delay directives are also respected, feeding into the rate limiter.
Respecting robots.txt is not just politeness—it is a contract between crawlers and site operators. Violating it can result in IP bans, legal issues, and exclusion from the content you are trying to index. A well-behaved crawler treats robots.txt violations as hard errors, not warnings.
HTTP redirects (301, 302, 307, 308) are common on the web. The crawler follows redirects up to a configurable maximum depth (default: 10) and records the entire redirect chain. The final resolved URL is used as the canonical URL for deduplication and indexing. Redirect loops are detected and broken by tracking visited URLs within each chain.
Raw HTML pages contain navigation menus, sidebars, footers, ads, cookie banners, and other boilerplate that is irrelevant to the actual content. The extraction pipeline uses a combination of heuristics and structural analysis to identify the main content area. It scores DOM subtrees by text density (ratio of text characters to HTML tag characters), link density (ratio of anchor text to total text), and structural signals (presence of <article>, <main>, or schema.org markup).
The text density heuristic exploits the fact that main content has a high ratio of visible text to HTML markup, while navigation and boilerplate have many tags relative to their text content. A DOM subtree with 80% text and 20% tags is almost certainly content; one with 20% text and 80% tags is almost certainly navigation.
SimHash produces a 64-bit fingerprint of each page’s extracted text. Pages whose SimHash values differ by fewer than 3 bits (Hamming distance < 3) are considered near-duplicates. The crawler maintains an in-memory index of fingerprints and only indexes the first occurrence. This catches duplicate content that URL-based deduplication misses, such as the same article served at /blog/post and /archive/2024/post.
SimHash is a locality-sensitive hash: similar documents produce similar hashes. By measuring the Hamming distance between fingerprints, the crawler can efficiently detect near-duplicates without comparing full document texts. This scales to millions of pages with constant memory per fingerprint (8 bytes).
After extraction, the clean text is split into overlapping chunks suitable for embedding and retrieval. Each chunk retains metadata about its source URL, page title, extraction timestamp, and position within the document. The chunks are then embedded using a vector model and indexed into a vector store for semantic search.
The crawl pipeline is structured as a set of concurrent workers connected by channels. This design allows each stage to operate at its own pace and be scaled independently.
// Crawler orchestrates the crawl pipeline.
type Crawler struct {
frontier *URLFrontier
fetchers int
extractors int
indexers int
robotsCache *RobotsCache
dedup *SimHashIndex
store DocumentStore
}
// CrawlConfig configures the crawl pipeline.
type CrawlConfig struct {
Seeds []string
AllowedDomains []string
MaxDepth int
FetchWorkers int
ExtractWorkers int
IndexWorkers int
RateLimit time.Duration
UserAgent string
}
// Run starts the crawl pipeline and blocks until complete.
func (c *Crawler) Run(ctx context.Context) error {
fetchQueue := make(chan *FetchRequest, 1000)
extractQueue := make(chan *FetchResponse, 500)
indexQueue := make(chan *ExtractedPage, 500)
var g errgroup.Group
// Frontier feeds fetch queue.
g.Go(func() error {
return c.frontier.Run(ctx, fetchQueue)
})
// Fetcher workers.
for i := 0; i < c.fetchers; i++ {
g.Go(func() error {
return c.fetchLoop(ctx, fetchQueue, extractQueue)
})
}
// Extractor workers.
for i := 0; i < c.extractors; i++ {
g.Go(func() error {
return c.extractLoop(ctx, extractQueue, indexQueue)
})
}
// Indexer workers.
for i := 0; i < c.indexers; i++ {
g.Go(func() error {
return c.indexLoop(ctx, indexQueue)
})
}
return g.Wait()
}// extractLoop processes fetched pages to extract content.
func (c *Crawler) extractLoop(
ctx context.Context,
in <-chan *FetchResponse,
out chan<- *ExtractedPage,
) error {
for {
select {
case <-ctx.Done():
return ctx.Err()
case resp, ok := <-in:
if !ok {
return nil
}
// Extract main content from HTML.
content, links := ExtractContent(resp.Body)
// Feed discovered links back to frontier.
for _, link := range links {
c.frontier.Enqueue(link, resp.Depth+1)
}
// Deduplication check.
fingerprint := SimHash(content)
if c.dedup.IsDuplicate(fingerprint) {
continue // Skip near-duplicate.
}
c.dedup.Add(fingerprint, resp.URL)
out <- &ExtractedPage{
URL: resp.URL,
Title: resp.Title,
Content: content,
Fingerprint: fingerprint,
FetchedAt: resp.FetchedAt,
}
}
}
}A crawler like this can build a comprehensive knowledge base from documentation sites, technical blogs, and API references. The crawled and extracted content is indexed into a vector store, providing the foundation for semantic search and question-answering over web content that would otherwise require manual curation.
By storing fetch timestamps and content fingerprints, the crawler can detect when documentation changes. Periodic re-crawls compare new fingerprints against stored ones, updating only the changed pages. This enables a documentation monitoring system that alerts teams when upstream dependencies change their APIs or deprecate features.
High-quality web content, once extracted and deduplicated, serves as training data for domain-specific models. The deduplication step is especially important here: duplicate content in training data causes models to overfit to repeated patterns and degrades generalization.
The URL frontier uses a priority queue to schedule fetches. Priority is determined by a scoring function that considers crawl depth (shallower pages are prioritized), domain diversity (avoid fetching too many pages from one domain consecutively), and staleness (pages not fetched recently get a boost). The frontier also enforces per-domain politeness by tracking the last fetch time for each domain.
// URLFrontier manages the queue of URLs to fetch.
type URLFrontier struct {
mu sync.Mutex
heap *priorityHeap
seen map[string]bool // Deduplicate URLs.
lastFetch map[string]time.Time // Per-domain last fetch time.
delay time.Duration // Minimum delay between fetches to same domain.
}
// Enqueue adds a URL to the frontier if not already seen.
func (f *URLFrontier) Enqueue(rawURL string, depth int) {
normalized := NormalizeURL(rawURL)
f.mu.Lock()
defer f.mu.Unlock()
if f.seen[normalized] {
return
}
f.seen[normalized] = true
priority := f.score(normalized, depth)
heap.Push(f.heap, &FrontierEntry{
URL: normalized,
Depth: depth,
Priority: priority,
})
}
func (f *URLFrontier) score(url string, depth int) float64 {
// Lower depth = higher priority. Domain diversity bonus.
depthScore := 1.0 / float64(depth+1)
return depthScore
}The SimHash algorithm works by hashing each word in the document, then combining the hashes into a single fingerprint by voting: for each bit position, if more word hashes have a 1 in that position than a 0, the fingerprint bit is 1, and vice versa. This produces a locality-sensitive fingerprint where documents sharing many words produce similar hashes.
// SimHash computes a 64-bit locality-sensitive hash of the text.
func SimHash(text string) uint64 {
var v [64]int
words := tokenize(text)
for _, word := range words {
h := fnv64(word)
for i := 0; i < 64; i++ {
if h&(1<<uint(i)) != 0 {
v[i]++
} else {
v[i]--
}
}
}
var fingerprint uint64
for i := 0; i < 64; i++ {
if v[i] > 0 {
fingerprint |= 1 << uint(i)
}
}
return fingerprint
}
// HammingDistance counts differing bits between two fingerprints.
func HammingDistance(a, b uint64) int {
return bits.OnesCount64(a ^ b)
}golang.org/x/net/html/charset to detect encoding from both headers and content.