ChromeDP is a Go package that provides a high-level API for controlling Chrome or Chromium browsers through the Chrome DevTools Protocol (CDP). Unlike tools like Selenium that require a separate WebDriver binary, ChromeDP communicates directly with Chrome over a WebSocket connection, giving you access to the full DevTools API: DOM manipulation, network interception, JavaScript execution, screenshot capture, PDF generation, performance profiling, and more. The “headless” mode runs Chrome without a visible window, making it suitable for server-side automation.
Headless browser automation is essential for tasks that require a full browser rendering engine: scraping JavaScript-rendered single-page applications, generating screenshots or PDFs of web pages, running end-to-end tests against web UIs, and extracting structured data from sites that use dynamic content loading. Unlike HTTP-only scrapers that parse raw HTML, a headless browser executes JavaScript, fires XHR requests, renders CSS, and produces the same DOM that a human user would see.
As a practical example, a content ingestion pipeline can use ChromeDP for web content ingestion: crawling web pages that require JavaScript execution, capturing page snapshots for indexing, and generating PDF archives of documentation sites. ChromeDP’s Go-native API integrates cleanly with Go’s concurrency model, allowing multiple browser tabs to operate in parallel using goroutines and context-based cancellation.
CDP is a JSON-RPC protocol that Chrome exposes over WebSocket. It provides domains (Page, DOM, Network, Runtime, Emulation, etc.), each containing methods (commands you send) and events (notifications Chrome sends). For example, Page.navigate is a method that loads a URL, and Page.loadEventFired is an event that fires when the page finishes loading. ChromeDP wraps these protocol-level details into idiomatic Go functions.
ChromeDP uses Go’s context.Context to manage browser, tab, and action lifecycles. The top-level context starts a Chrome process; child contexts create new tabs (browser contexts). Canceling a context closes the associated tab or browser. This maps perfectly to Go’s context hierarchy: a request handler creates a tab context with a timeout, runs actions, and the tab is automatically cleaned up when the context expires.
ChromeDP operations are expressed as actions—composable units of work that execute in sequence within a tab. chromedp.Navigate, chromedp.Click, chromedp.WaitVisible, and chromedp.Text are all actions. The chromedp.Run function executes a slice of actions in order, handling the CDP protocol communication, waiting for events, and propagating errors.
Most ChromeDP actions target DOM elements using CSS selectors or XPath expressions. Actions like chromedp.Click("#submit") implicitly wait for the element to appear in the DOM before acting. This built-in waiting eliminates the flaky timing issues that plague browser automation—you don’t need explicit sleeps. If the element never appears, the action fails with the context’s timeout.
Chrome’s browser context (incognito profile) provides full isolation: separate cookies, local storage, cache, and service workers. ChromeDP creates a new browser context for each chromedp.NewContext call. This means concurrent scraping sessions don’t share state—cookies set in one session don’t leak to another. Each browser context is a clean slate, equivalent to opening a fresh incognito window.
CDP’s Fetch and Network domains allow intercepting HTTP requests before they leave the browser. You can modify headers, block requests (e.g., skip loading images or ads for faster scraping), inject custom responses, or log all network traffic for debugging. This is implemented by enabling fetch.Enable and handling fetch.EventRequestPaused events in a goroutine.
The fundamental ChromeDP workflow: create a context (which starts Chrome), run a sequence of actions, and read the results:
package main
import (
"context"
"fmt"
"log"
"time"
"github.com/chromedp/chromedp"
)
func main() {
// Create a Chrome instance with a 30-second timeout
ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()
ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
defer cancel()
var title string
var body string
// Run a sequence of actions
err := chromedp.Run(ctx,
// Navigate to the page
chromedp.Navigate("https://example.com"),
// Wait for the page to be fully loaded
chromedp.WaitReady("body"),
// Extract the page title
chromedp.Title(&title),
// Extract text content from an element
chromedp.Text("body", &body, chromedp.NodeVisible),
)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Title: %s\n", title)
fmt.Printf("Body length: %d chars\n", len(body))
}ChromeDP can capture full-page screenshots and generate PDFs, which is useful for archiving web content or generating visual regression baselines:
func captureScreenshot(ctx context.Context, url string) ([]byte, error) {
var screenshot []byte
err := chromedp.Run(ctx,
chromedp.Navigate(url),
chromedp.WaitReady("body"),
// Capture viewport screenshot as PNG
chromedp.CaptureScreenshot(&screenshot),
)
return screenshot, err
}
func captureFullPageScreenshot(ctx context.Context, url string) ([]byte, error) {
var screenshot []byte
err := chromedp.Run(ctx,
chromedp.Navigate(url),
chromedp.WaitReady("body"),
// Full page screenshot captures the entire scrollable area
chromedp.FullScreenshot(&screenshot, 90), // 90% quality
)
return screenshot, err
}
func generatePDF(ctx context.Context, url string) ([]byte, error) {
var pdf []byte
err := chromedp.Run(ctx,
chromedp.Navigate(url),
chromedp.WaitReady("body"),
chromedp.ActionFunc(func(ctx context.Context) error {
var err error
pdf, _, err = page.PrintToPDF().
WithPrintBackground(true).
WithPreferCSSPageSize(true).
Do(ctx)
return err
}),
)
return pdf, err
}ChromeDP supports full interactive automation: filling forms, clicking buttons, handling dialogs, and executing arbitrary JavaScript:
func loginAndExtract(ctx context.Context, username, password string) (string, error) {
var dashboardContent string
err := chromedp.Run(ctx,
// Navigate to login page
chromedp.Navigate("https://app.example.com/login"),
// Wait for the login form
chromedp.WaitVisible("#login-form"),
// Fill in credentials
chromedp.SendKeys("#username", username),
chromedp.SendKeys("#password", password),
// Click the login button
chromedp.Click("#login-button"),
// Wait for redirect to dashboard
chromedp.WaitVisible("#dashboard"),
// Extract dashboard content
chromedp.Text("#dashboard", &dashboardContent),
)
return dashboardContent, err
}
func executeJS(ctx context.Context) (string, error) {
var result string
err := chromedp.Run(ctx,
chromedp.Navigate("https://example.com"),
chromedp.WaitReady("body"),
// Execute JavaScript and capture the return value
chromedp.Evaluate(`
(() => {
const links = document.querySelectorAll('a[href]');
return JSON.stringify(
Array.from(links).map(a => ({
text: a.textContent.trim(),
href: a.href
}))
);
})()
`, &result),
)
return result, err
}ChromeDP leverages Go’s concurrency primitives for parallel browser operations. Each tab runs in its own goroutine with its own context:
func scrapeURLs(parentCtx context.Context, urls []string, concurrency int) map[string]string {
results := make(map[string]string)
var mu sync.Mutex
sem := make(chan struct{}, concurrency)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
sem <- struct{}{} // acquire semaphore
go func(u string) {
defer wg.Done()
defer func() { <-sem }() // release semaphore
// Each goroutine gets its own tab (browser context)
tabCtx, cancel := chromedp.NewContext(parentCtx)
defer cancel()
tabCtx, cancel = context.WithTimeout(tabCtx, 15*time.Second)
defer cancel()
var content string
err := chromedp.Run(tabCtx,
chromedp.Navigate(u),
chromedp.WaitReady("body"),
chromedp.OuterHTML("html", &content),
)
if err != nil {
log.Printf("failed %s: %v", u, err)
return
}
mu.Lock()
results[u] = content
mu.Unlock()
}(url)
}
wg.Wait()
return results
}Chrome’s behavior is configured through command-line flags. ChromeDP allows setting these via chromedp.NewExecAllocator. Common flags include --disable-gpu (required in some headless environments), --no-sandbox (required in Docker/CI), --disable-web-security (for cross-origin testing), and --user-agent=... (for anti-detection). The allocator also controls the Chrome binary path, user data directory, and proxy settings.
In a content ingestion service, ChromeDP crawls web pages that rely on JavaScript rendering. Many modern documentation sites (React, Vue, Angular-based) serve a minimal HTML shell and render content client-side. An HTTP-only crawler sees an empty <div id="root">; ChromeDP sees the fully rendered page. The rendered DOM is extracted and fed into the search indexing pipeline, ensuring that JavaScript-rendered content is searchable.
ChromeDP captures pixel-perfect screenshots at specified viewport sizes. A visual regression test suite navigates to each page, captures a screenshot, and compares it against a baseline image using pixel-diff algorithms. This catches CSS regressions, layout shifts, and rendering bugs that functional tests miss. The Go test framework coordinates the process, and failed comparisons produce a diff image highlighting the changes.
For generating invoices, reports, or documentation PDFs, ChromeDP renders HTML templates in Chrome and exports them via Page.printToPDF. This produces high-fidelity PDFs with correct CSS rendering (including flexbox, grid, and web fonts) that server-side HTML-to-PDF libraries often handle poorly. Chrome’s print engine respects @page CSS rules for margins, headers, and page breaks.
Some data sources require browser-based authentication (OAuth flows, CAPTCHA challenges, multi-factor auth with TOTP). ChromeDP can automate the login process, maintain session cookies across requests, and extract data from authenticated pages. The session cookies can be extracted and reused for subsequent HTTP-only requests, combining browser automation for auth with faster HTTP clients for data extraction.
Starting a new Chrome process for each request is expensive (~500ms). A pool of pre-started Chrome instances serves requests with much lower latency. The pool manages lifecycle, health checks, and recycling of Chrome processes:
type ChromePool struct {
allocCtx context.Context
cancel context.CancelFunc
maxTabs int
sem chan struct{}
}
func NewChromePool(maxTabs int, opts ...chromedp.ExecAllocatorOption) *ChromePool {
// Merge default options with custom ones
allOpts := append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", true),
chromedp.Flag("disable-gpu", true),
chromedp.Flag("no-sandbox", true),
chromedp.WindowSize(1920, 1080),
)
allOpts = append(allOpts, opts...)
allocCtx, cancel := chromedp.NewExecAllocator(context.Background(), allOpts...)
return &ChromePool{
allocCtx: allocCtx,
cancel: cancel,
maxTabs: maxTabs,
sem: make(chan struct{}, maxTabs),
}
}
// Do acquires a tab from the pool, runs actions, and releases the tab.
func (p *ChromePool) Do(ctx context.Context, actions ...chromedp.Action) error {
// Acquire a semaphore slot
select {
case p.sem <- struct{}{}:
defer func() { <-p.sem }()
case <-ctx.Done():
return ctx.Err()
}
// Create a new browser context (tab) within the shared Chrome process
tabCtx, cancel := chromedp.NewContext(p.allocCtx)
defer cancel()
// Apply the caller's timeout
tabCtx, cancel = context.WithTimeout(tabCtx, getTimeout(ctx))
defer cancel()
return chromedp.Run(tabCtx, actions...)
}
func (p *ChromePool) Close() {
p.cancel()
}For scraping tasks, blocking unnecessary resources (images, fonts, analytics scripts) dramatically reduces page load time and bandwidth:
func scrapeWithBlocking(ctx context.Context, url string) (string, error) {
var content string
// Set up request interception to block heavy resources
err := chromedp.Run(ctx,
// Enable network interception via Fetch domain
chromedp.ActionFunc(func(ctx context.Context) error {
return fetch.Enable().
WithPatterns([]*fetch.RequestPattern{
{ResourceType: network.ResourceTypeImage, RequestStage: fetch.RequestStageRequest},
{ResourceType: network.ResourceTypeFont, RequestStage: fetch.RequestStageRequest},
{ResourceType: network.ResourceTypeMedia, RequestStage: fetch.RequestStageRequest},
}).Do(ctx)
}),
)
if err != nil {
return "", err
}
// Handle intercepted requests in a goroutine
chromedp.ListenTarget(ctx, func(ev interface{}) {
if e, ok := ev.(*fetch.EventRequestPaused); ok {
go func() {
// Block the request by failing it
_ = fetch.FailRequest(e.RequestID, network.ErrorReasonBlockedByClient).Do(ctx)
}()
}
})
err = chromedp.Run(ctx,
chromedp.Navigate(url),
chromedp.WaitReady("body"),
chromedp.OuterHTML("html", &content),
)
return content, err
}Websites detect headless browsers through JavaScript checks (navigator.webdriver, missing plugins, viewport anomalies). For legitimate scraping of sites that block bots, apply anti-detection patches:
func stealthAllocator() (context.Context, context.CancelFunc) {
opts := append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", true),
chromedp.Flag("disable-blink-features", "AutomationControlled"),
chromedp.UserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "+
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"),
chromedp.WindowSize(1920, 1080),
)
return chromedp.NewExecAllocator(context.Background(), opts...)
}
// Inject stealth scripts before page load
func injectStealth(ctx context.Context) chromedp.ActionFunc {
return func(ctx context.Context) error {
// Remove the webdriver property that reveals automation
_, err := page.AddScriptToEvaluateOnNewDocument(`
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3]});
Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en']});
window.chrome = { runtime: {} };
`).Do(ctx)
return err
}
}Browser automation is inherently unreliable—pages may load slowly, elements may not appear, JavaScript may error. Every ChromeDP operation should have a timeout via context.WithTimeout. Design your automation to degrade gracefully: if an optional element times out, log a warning and continue; if a critical element is missing, fail fast with a clear error. Never use arbitrary time.Sleep calls—use chromedp.WaitVisible or chromedp.WaitReady instead.
NewExecAllocator) and create browser contexts (tabs) within it, which is 10–50 ms. Only restart the process when it becomes unhealthy.chromedp.Evaluate calls are fast (~1 ms) for simple expressions but can be slow for complex DOM traversals on large pages. Batch multiple extractions into a single Evaluate call that returns a JSON object rather than making many individual calls.defer cancel() immediately after creating the context.chromedp.WaitVisible on the result element, or listen for network idle events with page.EventLifecycleEvent (name: “networkIdle”) to ensure asynchronous content has loaded.chrome processes.chromedp.Navigate succeeds as long as the request is dispatched, even if the page returns a 404 or 500 error. Check the HTTP status by listening to network.EventResponseReceived events or by examining the page content after navigation..css-1a2b3c) break when the page is rebuilt. Prefer data-testid attributes, semantic selectors (nav > a), or XPath expressions that target text content.