Web crawlers that re-fetch entire pages on every run waste bandwidth, hammer origin servers, and slow down ingestion pipelines. An HTTP-aware content cache sits between the crawler and the network, storing response bodies on disk while honoring the HTTP caching semantics that servers already communicate—Cache-Control directives, ETag validators, and Last-Modified timestamps. When a cached response is still fresh, the crawler skips the network entirely; when it’s stale, the cache performs a conditional request and only downloads the body if it actually changed.
This pattern is used in document ingestion pipelines. The cache is disk-backed, using a content-addressable store keyed by SHA-256 hashes of response bodies. This means that when multiple URLs serve identical content (common with CDN mirrors or canonicalized pages), the body is stored only once. An LRU index tracks access patterns and enforces a maximum cache size, evicting the least-recently-used entries when the disk budget is exceeded.
This design reduces crawl time by orders of magnitude on repeat runs, respects origin server wishes expressed through standard HTTP headers, and keeps disk usage bounded even when crawling millions of pages. Understanding how HTTP caching works at the protocol level—and how to implement it correctly in a disk-backed store—is essential for building production-grade content pipelines.
The Cache-Control header is the primary mechanism servers use to communicate caching policy. Directives like max-age=3600 specify how many seconds a response remains fresh without revalidation. no-cache requires revalidation on every access (but still allows storage), while no-store prohibits caching entirely. A correct cache implementation must parse these directives and respect their semantics—serving stale content or caching prohibited responses breaks both correctness and the implicit contract with the server.
HTTP caching distinguishes between freshness (can I use the cached response without contacting the server?) and validation (has the content changed since I last fetched it?). Fresh responses are served directly from cache. Stale responses trigger a conditional request using If-None-Match (ETag) or If-Modified-Since headers. A 304 Not Modified response confirms the cache is still valid without re-transferring the body.
Instead of keying stored bodies by URL, the cache computes a SHA-256 hash of each response body and uses that hash as the storage key. A separate metadata index maps URLs to their current body hash, response headers, and freshness timestamps. This indirection means identical bodies are stored once regardless of how many URLs point to them—a significant space savings when crawling sites that serve the same content at multiple paths (e.g., with and without trailing slashes, or across locale-prefixed URLs).
By hashing the response body and using the hash as the storage filename, the cache automatically deduplicates identical content. When URL /page and /page/ return the same HTML, both metadata entries point to the same hash file on disk. Deletion only removes the body file when its reference count drops to zero, preventing accidental data loss when one URL’s entry is evicted while another still references it.
The cache maintains an LRU (Least Recently Used) index to bound disk usage. Each cache hit promotes the entry to the head of the LRU list. When total cache size exceeds the configured maximum, entries at the tail are evicted—their metadata is removed and, if no other URL references the same body hash, the body file is deleted. The LRU index itself is persisted to disk so it survives process restarts.
Beyond LRU eviction for space management, the cache enforces TTLs derived from HTTP headers. An entry’s TTL is computed from Cache-Control: max-age, the Expires header, or a default TTL if neither is present. Expired entries are not immediately deleted—they remain on disk for conditional revalidation. Only entries that fail revalidation (the server returns a new body) have their old body replaced.
When a cached entry is stale, the cache sends a conditional GET request. If the original response included an ETag header, the cache sends If-None-Match: "etag-value". If it included Last-Modified, the cache sends If-Modified-Since: timestamp. The server responds with either 304 Not Modified (cache wins, no body transferred) or 200 OK with a new body (cache updates its stored copy). This mechanism is the key to efficient re-crawling—most pages don’t change between runs, so the vast majority of revalidations result in 304s.
If-None-Match or If-Modified-Sincepackage crawlcache
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"time"
)
// Entry represents a cached HTTP response.
type Entry struct {
URL string `json:"url"`
BodyHash string `json:"body_hash"` // SHA-256 hex digest
StatusCode int `json:"status_code"`
Headers http.Header `json:"headers"`
ETag string `json:"etag,omitempty"`
LastModified time.Time `json:"last_modified,omitempty"`
FetchedAt time.Time `json:"fetched_at"`
ExpiresAt time.Time `json:"expires_at"`
BodySize int64 `json:"body_size"`
}
// IsFresh returns true if the entry has not expired.
func (e *Entry) IsFresh(now time.Time) bool {
return now.Before(e.ExpiresAt)
}
// ComputeHash returns the SHA-256 hex digest of the given data.
func ComputeHash(data []byte) string {
h := sha256.Sum256(data)
return hex.EncodeToString(h[:])
}package crawlcache
import (
"context"
"io"
"net/http"
)
// FetchWithCache performs an HTTP GET, using cached data when possible.
func (c *Cache) FetchWithCache(ctx context.Context, url string) ([]byte, error) {
entry, found := c.index.Get(url)
if found && entry.IsFresh(c.clock.Now()) {
// Cache hit: serve from disk without network round-trip.
c.lru.Promote(url)
c.stats.Hits++
return c.store.ReadBody(entry.BodyHash)
}
// Build request, adding conditional headers if we have a stale entry.
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return nil, err
}
if found {
if entry.ETag != "" {
req.Header.Set("If-None-Match", entry.ETag)
} else if !entry.LastModified.IsZero() {
req.Header.Set("If-Modified-Since",
entry.LastModified.UTC().Format(http.TimeFormat))
}
}
resp, err := c.client.Do(req)
if err != nil {
// On network error, serve stale if available.
if found {
c.stats.StaleHits++
return c.store.ReadBody(entry.BodyHash)
}
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusNotModified && found {
// 304: content unchanged. Refresh TTL and return cached body.
entry.ExpiresAt = computeExpiry(resp.Header, c.clock.Now(), c.defaultTTL)
c.index.Put(url, entry)
c.lru.Promote(url)
c.stats.Revalidated++
return c.store.ReadBody(entry.BodyHash)
}
// Full response: read body, compute hash, store.
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
hash := ComputeHash(body)
if err := c.store.WriteBody(hash, body); err != nil {
return nil, err
}
newEntry := Entry{
URL: url,
BodyHash: hash,
StatusCode: resp.StatusCode,
Headers: resp.Header,
ETag: resp.Header.Get("ETag"),
FetchedAt: c.clock.Now(),
ExpiresAt: computeExpiry(resp.Header, c.clock.Now(), c.defaultTTL),
BodySize: int64(len(body)),
}
if lm := resp.Header.Get("Last-Modified"); lm != "" {
newEntry.LastModified, _ = http.ParseTime(lm)
}
c.index.Put(url, newEntry)
c.lru.Insert(url, newEntry.BodySize)
c.evictIfNeeded()
c.stats.Misses++
return body, nil
}In a typical document ingestion pipeline, the HTTP cache sits between the web source and the document processor. When re-ingesting a documentation site that has 10,000 pages but only 50 changed since the last run, the cache ensures only those 50 pages are re-downloaded. The remaining 9,950 are served from disk in microseconds instead of milliseconds-per-request over the network.
Respecting Cache-Control headers is a form of politeness—the server is telling you how often content changes. A crawler that ignores these signals and re-fetches everything on every run puts unnecessary load on origin servers. The cache translates HTTP semantics into real bandwidth savings without requiring the crawler logic to understand caching.
A populated cache enables fully offline development and testing of downstream pipeline stages. Developers can run the parsing, indexing, and search stages against cached content without network access, dramatically speeding up the development feedback loop.
TTL logic depends on the current time. Injecting a clock interface (func() time.Time) into the cache allows tests to control time progression, verifying that entries expire correctly and conditional requests fire at the right moment without waiting for real time to pass.
// Clock abstracts time for testability.
type Clock interface {
Now() time.Time
}
type realClock struct{}
func (realClock) Now() time.Time { return time.Now() }
type mockClock struct{ current time.Time }
func (m *mockClock) Now() time.Time { return m.current }
func (m *mockClock) Advance(d time.Duration) { m.current = m.current.Add(d) }
func TestCacheExpiry(t *testing.T) {
clk := &mockClock{current: time.Now()}
cache := NewCache(WithClock(clk), WithDefaultTTL(1*time.Hour))
// First fetch populates cache.
_, _ = cache.FetchWithCache(ctx, "https://example.com/page")
// Advance past TTL.
clk.Advance(2 * time.Hour)
// Next fetch should trigger a conditional request, not serve from cache.
// (Verified by checking stats or intercepting the HTTP request.)
}Because multiple URLs can reference the same body hash, the cache maintains a reference count for each stored body. When a URL entry is evicted or updated (body changed), the old hash’s reference count is decremented. The body file is only deleted when the count reaches zero. This prevents a subtle bug where evicting one URL deletes a body still referenced by another.
When the network is unavailable, the cache can serve stale content rather than failing outright. This “stale-while-error” behavior mirrors the stale-while-revalidate and stale-if-error Cache-Control extensions defined in RFC 5861. The crawler continues making progress with slightly outdated content, and entries are refreshed on the next successful run.
Vary headers: The Vary header indicates that the response varies based on request headers (e.g., Vary: Accept-Encoding). A cache that ignores Vary may serve a gzip-compressed body to a client that didn’t request compression. For crawling purposes, normalizing the Accept-Encoding header on all requests avoids this issue.no-store responses: The no-store directive means the response must not be written to disk at all—not even for conditional revalidation. Caching these responses violates the HTTP specification and can expose sensitive content. Always check for no-store before writing to the cache.singleflight.Group to coalesce concurrent requests for the same URL into a single network round-trip.Expires header: The Expires header uses an absolute timestamp from the server’s clock. If the client’s clock is significantly different, entries may appear fresh when they’re stale or vice versa. Prefer Cache-Control: max-age (relative) over Expires (absolute) when computing TTLs, and fall back to Expires only if max-age is absent.