Change Data Capture turns database writes into an ordered stream that downstream systems can consume. Instead of periodically dumping a whole database or writing every change twice from application code, a CDC pipeline reads the database replication log, emits row-level change events, stores offsets, and delivers those events to Kafka, Pulsar, Kinesis, a search index, a cache, or another database.
DDIA treats CDC as a core stream-processing pattern because it bridges systems of record and derived data. A relational database remains the place where transactions commit. The CDC stream carries the committed facts outward so other systems can build projections, indexes, analytics tables, machine-learning features, audit trails, or materialized views without participating in the original transaction.
Log-based CDC is different from trigger-based sync. Triggers are application-visible database objects that run on every write path and can be hard to deploy uniformly. Log-based CDC uses replication infrastructure the database already needs for recovery and replicas: PostgreSQL WAL with logical decoding, MySQL binlog, MongoDB oplog, SQL Server change tables, Oracle redo, or similar mechanisms. Debezium popularized this architecture by packaging database-specific log readers as Kafka Connect source connectors.
The hard part is not simply reading bytes from a log. A production CDC system must coordinate an initial consistent snapshot with a later stream position, encode change events with schemas, preserve per-key or per-partition order, handle deletes and primary-key updates, store offsets transactionally enough for restart, manage WAL/binlog retention, and make downstream consumers idempotent because at-least-once delivery remains the baseline guarantee.
A CDC consumer normally needs a full copy of current state and every later change. The safe bootstrap is therefore two-part: take a consistent snapshot at a known log position, then stream all committed changes after that position. Without coordination, the pipeline can miss writes that happen during the snapshot or apply the same change in the wrong order.
The connector records the source database position for every emitted event: a PostgreSQL LSN, a MySQL binlog filename and offset, or an equivalent source-specific cursor. On restart, it resumes from the last stored position. In Kafka Connect deployments, connector offsets are stored in Kafka internal topics and committed as part of the source connector runtime protocol.
A CDC event usually contains a key, the previous row image when available, the new row image, operation type, timestamp, source partition, and source offset. Sinks can consume the full envelope for auditing or flatten it into the latest row state with a transformation such as Debezium’s new-record-state extraction.
The database log is already the serialization point for committed changes. CDC makes that serialization point available outside the database, so derived systems can replay changes in commit order rather than racing against snapshots, polling queries, and dual writes.
Debezium documents at-least-once delivery as the baseline: no committed change should be missed, but a record can be delivered more than once after crashes, rebalances, or uncertain offset commits. Exactly-once source delivery depends on Kafka Connect and Kafka transaction support and still deserves careful validation.
A connector runs beside the source database, not inside application code. It connects as a replication client, decodes log records into table-oriented events, serializes them, sends them to a durable log such as Kafka, and records the source offset after records are acknowledged.
type SourceOffset struct {
ServerID string
LSN string
}
type ChangeEvent struct {
Key []byte
Before map[string]any
After map[string]any
Operation string
Source SourceOffset
}
func streamChanges(ctx context.Context, db LogicalReplication, producer Producer, offsets OffsetStore) error {
offset, err := offsets.Load(ctx)
if err != nil {
return err
}
stream, err := db.StartReplication(ctx, offset.LSN)
if err != nil {
return err
}
for stream.Next(ctx) {
tx := stream.Transaction()
batch := make([]ChangeEvent, 0, len(tx.Changes))
for _, change := range tx.Changes {
event := decodeRowChange(change)
event.Source = SourceOffset{ServerID: db.ServerID(), LSN: tx.CommitLSN}
batch = append(batch, event)
}
if err := producer.SendBatch(ctx, topicFor(tx), batch); err != nil {
return err
}
if err := offsets.Store(ctx, SourceOffset{ServerID: db.ServerID(), LSN: tx.CommitLSN}); err != nil {
return err
}
db.Acknowledge(ctx, tx.CommitLSN)
}
return stream.Err()
}For PostgreSQL, Debezium describes a snapshot workflow that starts a transaction, reads the current transaction-log position, scans tables, commits the snapshot transaction, records snapshot completion, and then streams from the captured log position. This preserves the invariant that no update committed during the snapshot is omitted.
func initialSnapshot(ctx context.Context, db Database, out Producer, offsets OffsetStore) error {
tx, err := db.BeginSnapshot(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)
start := tx.CurrentLogPosition()
for _, table := range tx.CapturedTables() {
rows := tx.Scan(table)
for rows.Next() {
event := ChangeEvent{
Key: primaryKey(rows.Row()),
After: rows.Row(),
Operation: "read",
Source: SourceOffset{ServerID: db.ServerID(), LSN: start},
}
if err := out.Send(ctx, topicForTable(table), event); err != nil {
return err
}
}
}
if err := tx.Commit(ctx); err != nil {
return err
}
return offsets.StoreSnapshotComplete(ctx, start)
}Consumers should apply events idempotently. A sink can store the latest source offset per destination row, reject stale events, upsert inserts and updates, and turn deletes into tombstones. This is what makes replay safe after a crash or backfill.
A common pipeline streams product, order, user, or document changes into Elasticsearch, OpenSearch, Redis, or a serving cache. The transactional database remains the source of truth. The derived store can be rebuilt from a snapshot plus the retained stream after mapping bugs or index corruption.
CDC lets one service publish committed state changes without adding fragile dual writes to request handlers. The outbox pattern tightens this further: the service writes business rows and outbox rows in the same database transaction, and CDC publishes only the outbox table as integration events.
Operational changes can feed lakehouse tables, stream processors, or feature stores with low latency. Kleppmann’s Bottled Water article frames this as extracting a consistent snapshot and then a real-time stream so downstream systems stay almost up to date while preserving write order.
Store source offsets with sink rows and compare incoming offsets before applying changes. The exact comparison is source-specific: PostgreSQL LSNs are ordered within a replication stream; MySQL offsets need binlog file ordering plus byte position; some connectors include transaction IDs and event numbers.
func applyCDCEvent(db *sql.DB, e ChangeEvent) error {
return withTx(db, func(tx *sql.Tx) error {
current := loadAppliedOffset(tx, e.Key)
if current != nil && compareOffsets(*current, e.Source) >= 0 {
return nil
}
switch e.Operation {
case "delete":
if err := deleteDestinationRow(tx, e.Key); err != nil {
return err
}
case "create", "update", "read":
if err := upsertDestinationRow(tx, e.Key, e.After); err != nil {
return err
}
}
return storeAppliedOffset(tx, e.Key, e.Source)
})
}The outbox pattern converts internal state changes into explicit integration events without publishing from the request thread. Application code writes an outbox row in the same transaction as the business update. CDC reads the outbox table and a router publishes compact event messages.
BEGIN;
UPDATE orders
SET status = 'paid', paid_at = now()
WHERE id = $1 AND status = 'pending';
INSERT INTO outbox_events(id, aggregate_id, event_type, payload, created_at)
VALUES ($2, $1, 'OrderPaid', $3, now());
COMMIT;CDC events should carry schema identifiers or use a registry. Bottled Water used Avro schemas generated from PostgreSQL tables and registered those schemas so consumers could decode typed records. Without schema discipline, rolling database migrations can break downstream consumers long after the source deployment looks successful.