Fsnotify Debounced Watch Service

Overview

File system watchers are the backbone of live-reload development servers, search index updaters, and backup synchronizers. The operating system provides low-level primitives — inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows — that notify a process when files change. However, these raw events are noisy: a single file save from an editor can produce multiple events (write, chmod, rename-swap), and batch operations like git checkout generate a storm of notifications that would overwhelm any handler processing them individually.

A well-designed search indexer solves this with a debounced watch service built on Go’s fsnotify library. The service recursively registers watched directories, assigns stable watch_dir_id values for tracking, and debounces events per file path with a 200-millisecond window. Only after a path has been quiet for 200ms does the service schedule an IndexFile or DeleteFile job. This means rapid successive writes to the same file produce exactly one indexing job, and background scans only fire once writes settle.

This pattern is broadly applicable beyond search indexing. Any system that reacts to file changes — asset pipelines, code generators, test runners, deployment watchers — benefits from the same debounce-and-dispatch architecture to avoid redundant work and race conditions with partially-written files.

Core Concepts

How It Works

The watch service runs as a long-lived goroutine that reads from the fsnotify event channel, maintains per-path debounce timers, and dispatches jobs to a work queue when timers fire.

Debounced Watcher Implementation

package watcher

import (
    "log"
    "os"
    "path/filepath"
    "sync"
    "time"

    "github.com/fsnotify/fsnotify"
)

const debounceDelay = 200 * time.Millisecond

// WatchService monitors directories and dispatches indexing jobs.
type WatchService struct {
    watcher    *fsnotify.Watcher
    timers     map[string]*time.Timer
    mu         sync.Mutex
    jobChan    chan Job
    dirIDs     map[string]int64  // path -> stable watch_dir_id
}

type Job struct {
    Type    string // "IndexFile" or "DeleteFile"
    Path    string
    DirID   int64
}

func New(jobChan chan Job) (*WatchService, error) {
    w, err := fsnotify.NewWatcher()
    if err != nil {
        return nil, err
    }
    return &WatchService{
        watcher: w,
        timers:  make(map[string]*time.Timer),
        jobChan: jobChan,
        dirIDs:  make(map[string]int64),
    }, nil
}

// AddRoot recursively watches a directory tree.
func (ws *WatchService) AddRoot(root string, dirID int64) error {
    ws.dirIDs[root] = dirID
    return filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
        if err != nil {
            return err
        }
        if d.IsDir() {
            return ws.watcher.Add(path)
        }
        return nil
    })
}

Event Loop with Debouncing

// Run starts the event processing loop. Blocks until ctx is cancelled.
func (ws *WatchService) Run() {
    for {
        select {
        case event, ok := <-ws.watcher.Events:
            if !ok {
                return
            }
            ws.handleEvent(event)

        case err, ok := <-ws.watcher.Errors:
            if !ok {
                return
            }
            log.Printf("watcher error: %v", err)
        }
    }
}

func (ws *WatchService) handleEvent(event fsnotify.Event) {
    path := event.Name

    // If a new directory is created, register it for watching.
    if event.Has(fsnotify.Create) {
        if info, err := os.Stat(path); err == nil && info.IsDir() {
            ws.watcher.Add(path)
            return
        }
    }

    // Determine job type from event.
    var jobType string
    switch {
    case event.Has(fsnotify.Remove) || event.Has(fsnotify.Rename):
        jobType = "DeleteFile"
    case event.Has(fsnotify.Create) || event.Has(fsnotify.Write):
        jobType = "IndexFile"
    default:
        return // Ignore chmod-only events.
    }

    // Debounce: reset the timer for this path.
    ws.mu.Lock()
    defer ws.mu.Unlock()

    if t, exists := ws.timers[path]; exists {
        t.Stop()
    }

    dirID := ws.findDirID(path)
    ws.timers[path] = time.AfterFunc(debounceDelay, func() {
        ws.mu.Lock()
        delete(ws.timers, path)
        ws.mu.Unlock()

        ws.jobChan <- Job{
            Type:  jobType,
            Path:  path,
            DirID: dirID,
        }
    })
}

// findDirID walks up from path to find the nearest registered root.
func (ws *WatchService) findDirID(path string) int64 {
    dir := filepath.Dir(path)
    for dir != "/" {
        if id, ok := ws.dirIDs[dir]; ok {
            return id
        }
        dir = filepath.Dir(dir)
    }
    return 0
}

Consumer Side: Processing Jobs

func processJobs(jobChan chan watcher.Job, indexer *search.Indexer) {
    for job := range jobChan {
        switch job.Type {
        case "IndexFile":
            content, err := os.ReadFile(job.Path)
            if err != nil {
                log.Printf("skip %s: %v", job.Path, err)
                continue
            }
            if err := indexer.Index(job.Path, job.DirID, content); err != nil {
                log.Printf("index error %s: %v", job.Path, err)
            }

        case "DeleteFile":
            if err := indexer.Delete(job.Path, job.DirID); err != nil {
                log.Printf("delete error %s: %v", job.Path, err)
            }
        }
    }
}

Practical Applications

A local search indexer uses this debounced watcher to keep its index in sync with the filesystem in near-real-time. When a user saves a document, the watcher detects the change, waits for the editor to finish writing (the debounce window), and dispatches an IndexFile job that re-parses and re-indexes the file. Search results reflect the change within a fraction of a second of the save completing.

The stable watch_dir_id system allows users to add and remove watched directories through the application’s UI. Each directory gets a persistent ID stored in SQLite. When a user removes a watched directory, the indexer can efficiently delete all entries with that watch_dir_id from the index rather than scanning the index for matching file paths.

The same debounce pattern appears across many development toolchains: build systems use file watching with debouncing for incremental rebuilds, and frontend dev servers debounce file changes before triggering hot module replacement. The 200ms window is a widely-used constant that works well across these use cases.

Implementation Patterns

Coalescing Rename Events

Editors like Vim and Emacs save files using an atomic rename: write to a temporary file, then rename it over the original. This produces a Rename event (old name disappearing) followed by a Create event (new name appearing). The debouncer naturally coalesces these into a single IndexFile job because the Create event resets the timer before the Rename timer fires.

Ignoring Patterns

Not all files should trigger indexing. Hidden files, build artifacts, and version control directories should be filtered out. Implement a path filter that runs before debouncing so ignored events don’t consume timer resources.

var ignorePatterns = []string{
    ".git", ".DS_Store", "node_modules",
    "*.swp", "*.tmp", "*~",
}

func shouldIgnore(path string) bool {
    base := filepath.Base(path)
    for _, pattern := range ignorePatterns {
        if matched, _ := filepath.Match(pattern, base); matched {
            return true
        }
    }
    return false
}

Batch Flush on Shutdown

When the service shuts down, all pending debounce timers should either be flushed (fire immediately) or cancelled. Flushing ensures that recent writes are indexed before the service stops; cancelling avoids wasted work if the service is restarting immediately and will do a full rescan.

Common Pitfalls

inotify Watch Limits

On Linux, each watched directory consumes an inotify watch descriptor. The default limit (fs.inotify.max_user_watches) is often 8192, which large directory trees can exhaust. When the limit is hit, fsnotify silently fails to watch new directories. Monitor the watch count and increase the system limit or selectively watch only directories that matter.

Reading Partially-Written Files

Even with debouncing, a 200ms window might not be enough for very large file writes. If an application writes a multi-gigabyte file, the debounce timer fires while the write is still in progress, and the indexer reads a truncated file. For large files, consider checking file size stability (size unchanged between two checks separated by the debounce window) before dispatching the job.

Symlink Loops

Recursive directory walking follows symlinks by default in some implementations. A symlink that points to a parent directory creates an infinite loop. Use filepath.WalkDir (which does not follow symlinks) rather than filepath.Walk, and explicitly resolve symlinks only when the target is outside the watched tree.

Event Ordering Guarantees

Filesystem notification APIs do not guarantee strict ordering of events across different paths. Two events for different files may arrive in any order relative to each other. The debouncer handles this correctly because each path has its own independent timer, but downstream job processors should not assume that jobs arrive in filesystem-operation order.

Further Reading