Web crawling at scale demands more than sending HTTP requests and parsing HTML. Production crawlers must normalize URLs to avoid duplicate fetches, respect robots.txt directives, rate-limit requests to avoid overwhelming target servers, handle transient failures with retries, and cache results to minimize redundant network traffic. The example below wraps the Colly web scraping framework with a WebSource layer that enforces all of these controls in a single, composable configuration.
Colly is a popular Go framework built on top of net/http and goquery that provides a callback-driven model for web scraping. However, Colly’s default configuration leaves many production concerns to the user. The WebSource crawl controls shown below add a URL normalizer that lowercases hostnames, strips fragment identifiers, and removes tracking parameters. A retrying transport wraps the HTTP client to handle 429 and 5xx responses with exponential backoff. Sitemap-driven crawl scheduling discovers pages automatically, and configurable TTLs on cached responses prevent stale data from persisting indefinitely.
Together, these controls turn a basic Colly collector into a well-behaved, production-grade crawler that can be pointed at documentation sites, API references, and knowledge bases to feed content into a document processing pipeline.
URL normalization ensures that semantically identical URLs are treated as a single resource. Without normalization, https://Docs.Example.Com/page#section and https://docs.example.com/page would be crawled as two separate pages. The normalizer applies a chain of transformations: lowercasing the scheme and hostname, removing default ports (80 for HTTP, 443 for HTTPS), stripping fragment identifiers, sorting query parameters alphabetically, and removing known tracking parameters like utm_source.
URL normalization produces a canonical form for each URL, ensuring that the crawler treats HTTPS://Example.COM/path?b=2&a=1#frag and https://example.com/path?a=1&b=2 as the same resource. This prevents duplicate fetches and duplicate entries in the document index.
The robots.txt protocol lets site operators specify which paths crawlers may and may not access. Colly has built-in support for parsing robots.txt, but the WebSource layer ensures this is always enabled and correctly configured with the appropriate user-agent string. Disallowed paths are filtered before any HTTP request is made, saving bandwidth and respecting site operator intent.
Aggressive crawling can trigger rate limits, get the crawler’s IP blocked, or degrade the target site’s performance for real users. The WebSource configures Colly’s LimitRule to enforce a minimum delay between requests to the same domain, with optional random jitter to avoid synchronized bursts. The delay and parallelism limits are configurable per domain pattern.
Rate limiting is not just about avoiding bans—it is about being a good citizen of the web. A “politeness budget” balances crawl throughput against impact on the target server. The WebSource defaults to 1 request per second per domain with 2 concurrent connections, which is appropriate for documentation sites.
Transient HTTP errors (429 Too Many Requests, 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable) should not cause a crawl to fail. The retrying transport wraps Go’s http.RoundTripper interface to intercept these responses and retry with exponential backoff. The transport respects Retry-After headers when present, falling back to a configurable base delay with jitter.
Rather than relying solely on link following, the WebSource can seed a crawl from the site’s sitemap.xml. This ensures that orphaned pages (those not linked from any other page) are still discovered. The sitemap parser handles both sitemap index files and individual sitemaps, extracting URLs and their lastmod dates to support incremental crawling.
Using sitemap.xml as a crawl seed guarantees complete coverage of a site’s published pages. Combined with lastmod timestamps, the crawler can skip pages that haven’t changed since the last crawl, significantly reducing unnecessary fetches.
The WebSource is initialized with a configuration struct that specifies target domains, rate limits, cache TTLs, and retry parameters. During initialization, the WebSource creates a Colly collector with the appropriate settings and wraps the HTTP transport.
WebSourceConfig with target URLs, allowed domains, rate limit rules, and caching options.// WebSourceConfig configures crawl behavior.
type WebSourceConfig struct {
AllowedDomains []string // Domains the crawler may visit.
StartURLs []string // Seed URLs for the crawl.
SitemapURLs []string // Sitemap URLs for discovery.
RateLimit time.Duration // Minimum delay between requests per domain.
Parallelism int // Max concurrent requests per domain.
MaxRetries int // Retries for transient errors.
CacheTTL time.Duration // How long cached responses are valid.
UserAgent string // User-Agent header value.
RespectRobots bool // Whether to obey robots.txt.
}
// NewWebSource creates a configured Colly-based web crawler.
func NewWebSource(cfg WebSourceConfig) (*WebSource, error) {
c := colly.NewCollector(
colly.AllowedDomains(cfg.AllowedDomains...),
colly.UserAgent(cfg.UserAgent),
colly.URLFilters(normalizeFilter()),
)
// Enable robots.txt.
if cfg.RespectRobots {
c.AllowURLRevisit = false
}
// Rate limiting.
c.Limit(&colly.LimitRule{
DomainGlob: "*",
Delay: cfg.RateLimit,
Parallelism: cfg.Parallelism,
RandomDelay: cfg.RateLimit / 2,
})
// Retrying transport.
c.WithTransport(NewRetryTransport(
http.DefaultTransport,
cfg.MaxRetries,
))
return &WebSource{collector: c, config: cfg}, nil
}A primary use case for this pattern is crawling documentation sites to feed a document processing pipeline. The WebSource crawls sites like API references, developer guides, and knowledge bases, extracting HTML content that is then converted to structured documents for indexing in a search engine. The crawl controls ensure that the ingestion is complete (via sitemap seeding), polite (via rate limiting), and resilient (via retries and caching).
By storing the lastmod timestamps from sitemaps and the ETag or Last-Modified headers from HTTP responses, the WebSource can detect when pages have changed since the last crawl. This enables incremental re-indexing: only modified pages are re-processed, keeping the document index fresh without re-crawling the entire site.
A single crawl job can target multiple sites by providing multiple start URLs and allowed domains. Each domain gets its own rate limit bucket, so a slow site does not block crawling of faster sites. This is used to aggregate documentation from multiple microservices into a unified knowledge base.
The URL normalizer is implemented as a chain of small, composable functions, each responsible for a single transformation. New normalizations (like removing a site-specific tracking parameter) can be added by appending a function to the chain without modifying existing ones.
// NormalizeFunc transforms a URL into its canonical form.
type NormalizeFunc func(u *url.URL) *url.URL
// NormalizeChain applies a sequence of normalizers.
func NormalizeChain(fns ...NormalizeFunc) NormalizeFunc {
return func(u *url.URL) *url.URL {
for _, fn := range fns {
u = fn(u)
}
return u
}
}
// DefaultNormalizer returns the standard normalization chain.
func DefaultNormalizer() NormalizeFunc {
return NormalizeChain(
LowercaseHost,
StripDefaultPort,
StripFragment,
SortQueryParams,
StripTrackingParams,
)
}
// LowercaseHost lowercases the hostname.
func LowercaseHost(u *url.URL) *url.URL {
u.Host = strings.ToLower(u.Host)
return u
}
// StripFragment removes the fragment identifier.
func StripFragment(u *url.URL) *url.URL {
u.Fragment = ""
return u
}The retrying transport implements http.RoundTripper, making it a drop-in replacement for the default transport. It inspects the response status code and retries on transient errors with exponential backoff. The Retry-After header is respected when available.
// RetryTransport wraps an http.RoundTripper with retry logic.
type RetryTransport struct {
base http.RoundTripper
maxRetries int
baseDelay time.Duration
}
func (t *RetryTransport) RoundTrip(req *http.Request) (*http.Response, error) {
var resp *http.Response
var err error
for attempt := 0; attempt <= t.maxRetries; attempt++ {
resp, err = t.base.RoundTrip(req)
if err != nil {
return nil, err // Network error, don't retry.
}
if !isRetryable(resp.StatusCode) {
return resp, nil
}
delay := t.backoffDelay(attempt, resp)
time.Sleep(delay)
resp.Body.Close()
}
return resp, nil // Return last response after exhausting retries.
}
func isRetryable(code int) bool {
return code == 429 || code == 500 || code == 502 || code == 503
}
func (t *RetryTransport) backoffDelay(attempt int, resp *http.Response) time.Duration {
if ra := resp.Header.Get("Retry-After"); ra != "" {
if secs, err := strconv.Atoi(ra); err == nil {
return time.Duration(secs) * time.Second
}
}
delay := t.baseDelay * time.Duration(1<<uint(attempt))
jitter := time.Duration(rand.Int63n(int64(delay / 2)))
return delay + jitter
}