Lambda and Kappa architectures are competing patterns for building derived data systems from immutable input streams. Both start from the same DDIA idea: operational systems produce facts, and downstream systems derive indexes, analytics tables, recommendations, counters, and machine-learning features from those facts. The disagreement is how many processing paths should compute those derived views.
Lambda architecture keeps two paths. A batch layer recomputes correct views from the complete historical data set. A speed layer computes recent updates with low latency. A serving layer merges the batch view with the real-time view at query time. The pattern became popular because early batch systems were reliable and scalable, while early stream processors were lower-latency but less trusted for complex stateful computation.
Kappa architecture removes the separate batch path. It stores the immutable log as the canonical data set and uses one stream processing program for both live updates and historical reprocessing. When logic changes, the team replays retained input through a new version of the stream job into a new serving table or topic. The trade-off shifts from maintaining two code paths to retaining enough log history and making the stream processor robust enough for full rebuilds.
The best architecture is rarely a slogan. Lambda is useful when batch and stream engines have materially different strengths, when historical recomputation is too heavy for the live stream path, or when offline data sources do not naturally fit a log. Kappa is attractive when the event log is complete, the transformation can run as one streaming program, and operational simplicity matters more than optimizing separate batch and speed systems.
Both architectures depend on retaining raw input events in an append-only form. If derived views are wrong, the system can replay original facts rather than trying to patch corrupted projections. This is the useful part of lambda that Jay Kreps preserves in his critique: transformation stages become reproducible materializations from stable inputs.
The batch layer periodically scans all historical input and writes a complete, correct view. In Hadoop-era systems, this might mean daily MapReduce or Spark jobs over HDFS. The output is slower to update but easier to reason about because each run sees a bounded data set and can recompute from scratch.
The speed layer consumes new events as they arrive and produces low-latency approximate or incremental views. It fills the freshness gap until the next batch view catches up. The speed layer is operationally expensive because it must duplicate some transformation logic in a different runtime and reconcile its output with the batch result.
Kappa keeps one processing path. Reprocessing means starting a new consumer group or job from the beginning of a retained log, writing to a new output location, validating it, and switching readers after it catches up. The log replaces the batch master data set when it is complete enough and retained long enough.
Whether the system is lambda or kappa, derived tables are not the source of truth. They are caches, indexes, projections, or serving views that should be disposable when code, schemas, or data quality rules change.
A lambda pipeline writes every input event to durable master storage. The batch job recomputes a base view over the full history. The speed job processes recent events that are not yet represented in the batch view. Queries read both outputs and merge them, often by taking the batch value plus a real-time delta.
type LambdaQuery struct {
Batch BatchView
Speed RealtimeView
Cutoff time.Time
}
func (q LambdaQuery) UserScore(userID string, now time.Time) (Score, error) {
base, err := q.Batch.ScoreAsOf(userID, q.Cutoff)
if err != nil {
return Score{}, err
}
delta, err := q.Speed.ScoreDelta(userID, q.Cutoff, now)
if err != nil {
return Score{}, err
}
return base.Add(delta), nil
}A kappa pipeline writes input to an append-only log and has one stream processor build the serving view. For a code change, it launches a second processor version from the beginning of the log. After the new projection catches up and validation passes, traffic switches to the new view. The old view can remain as rollback until confidence is high.
func rebuildProjection(log EventLog, output ProjectionStore, version string) error {
consumer := log.NewConsumer(ConsumerOptions{
GroupID: "projection-" + version,
StartAt: EarliestOffset,
ReadThrough: LatestOffset,
})
for consumer.Next() {
event := consumer.Event()
update := transform(version, event)
if err := output.Apply(version, event.ID, update); err != nil {
return err
}
}
return output.MarkReady(version, consumer.LastOffset())
}Cutover is a data migration problem. Readers need to know which projection version is active, whether the new version has caught up to the live tail, and how to roll back. Teams often dual-run old and new projections, compare counts and samples, then flip a small routing table rather than changing every consumer.
Lambda fits systems that need heavy offline models plus fresh user activity. A batch layer can train or recompute expensive features over a complete history, while a speed layer adds recent clicks, impressions, and inventory changes. The serving layer combines stable offline features with low-latency deltas.
Kappa is often enough for search and document projections. The canonical log contains create, update, and delete events; a stream processor updates Elasticsearch, OpenSearch, Solr, or a database table; and a rebuild creates a replacement index from the same log when mappings or analyzers change.
Metrics pipelines may use lambda when historical corrections and backfills are common, because the batch layer can rebuild windows from late data. Fraud pipelines often prefer kappa when low latency and one consistent rule implementation matter more than separate offline recomputation.
Modern lakehouse systems blur the boundary. A table format such as Iceberg, Delta Lake, or Hudi may be updated by streaming jobs and compacted by batch jobs. The architectural question remains the same: is there one logical transformation that can be replayed, or are there intentionally separate offline and online computations?
Do not overwrite the only serving table during a rebuild. Write projection versions side by side, store build metadata, and switch readers through a small indirection point. This turns reprocessing into a reversible deployment instead of a destructive migration.
CREATE TABLE projection_versions (
name text PRIMARY KEY,
status text NOT NULL CHECK (status IN ('building', 'ready', 'active', 'retired')),
high_watermark text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE user_scores_v42 (
user_id text PRIMARY KEY,
score numeric NOT NULL,
source_offset text NOT NULL
);
UPDATE projection_versions
SET status = CASE
WHEN name = 'user_scores_v42' THEN 'active'
WHEN status = 'active' THEN 'retired'
ELSE status
END
WHERE name IN ('user_scores_v41', 'user_scores_v42');The canonical stream should preserve facts, not pre-joined, UI-shaped blobs. Store stable identifiers, event time, source offsets, schema versions, and enough payload to recompute later. If the canonical stream omits a field needed for a future derivation, kappa replay cannot recover it without going back to another source.
Track input size, replay throughput, output write rate, and catch-up lag before relying on kappa. A log that takes 20 days to replay cannot support weekly algorithm iteration unless the job can scale out, the output can absorb backfill writes, or snapshots are introduced as restart points.
A kappa system can still use compacted snapshots to avoid replaying years of events for every rebuild. The important distinction is that the same transformation semantics remain in one path. Snapshots are optimization artifacts, not a separate batch implementation with different business rules.