Modern stream processing is not one architecture. Apache Flink, Spark Structured Streaming, Apache Beam, and Kafka Streams all process unbounded data, but they make different commitments about latency, state, deployment, portability, SQL support, failure recovery, and operational ownership. Choosing between them by popularity usually produces a system that is either too expensive, too slow, or too hard to operate for the workload.
DDIA frames stream processing as the construction of derived data systems from event logs. That framing leads to the real selection question: what kind of derived state are you building, how fresh must it be, how often will it be corrected, how large is the state, and who owns the runtime? A fraud detector, hourly data lake aggregation, embedded service enrichment, and cross-cloud portable pipeline can all be “streaming” applications while needing different engines.
Flink is a native event-at-a-time streaming engine with strong event-time tools, keyed state, checkpoints, and low-latency stateful processing. Spark Structured Streaming treats a stream as an incrementally maintained table and favors SQL, batch integration, and micro-batch throughput. Beam is a programming model and portability layer that separates pipeline semantics from runner choice. Kafka Streams is a library embedded in an application, using Kafka partitions, local state stores, and changelog topics instead of a separate cluster manager.
This topic is the comparison layer after learning each engine individually. It teaches a decision framework, not a ranking. The best engine is the one whose constraints match the workload’s latency tier, state model, source and sink semantics, team skills, and failure budget.
The execution model controls latency and semantics. Flink runs long-lived operators that process records continuously. Spark Structured Streaming usually runs micro-batches and can also run a narrower continuous mode. Beam describes a pipeline graph and delegates execution to a runner such as Dataflow, Flink, or Spark. Kafka Streams runs a processor topology inside ordinary application instances.
Stateful streaming is where engine differences become expensive. Flink exposes keyed state and snapshots it through checkpoints. Spark stores streaming state under checkpoint directories and updates it per trigger. Kafka Streams keeps task-local state stores, commonly RocksDB, backed by Kafka changelog topics. Beam exposes state and timers in the programming model, but the concrete behavior depends on the runner.
Event time, watermarks, windows, triggers, and late data are first-class in Flink and Beam. Spark Structured Streaming supports event-time watermarks and windows, but micro-batch trigger cadence still shapes output latency. Kafka Streams supports event-time processing and windows, but it is most natural when stream partitioning and local state align with Kafka topics.
Flink and Spark normally require a cluster or managed service. Beam may hide the runtime behind a managed runner, but the runner still has operational limits. Kafka Streams is just a library in your service, so deployment is familiar, but the application team now owns rebalances, state directory sizing, changelog topics, standby replicas, and partition-count planning.
Changing engines can change output timing, late-data behavior, state recovery, and exactly-once boundaries. A stream-processing choice is not merely an implementation detail once downstream systems depend on its update cadence and correction semantics.
Start with non-negotiable workload constraints. Latency, state size, event-time correctness, source replayability, sink idempotency, team runtime ownership, and language ecosystem usually narrow the field quickly. The matrix below is deliberately operational rather than marketing-oriented.
def choose_stream_engine(workload):
if workload.needs_portable_pipeline_api:
return "Apache Beam with a runner validated against required features"
if workload.embedded_in_service and workload.sources_are_kafka_only:
if workload.parallelism <= workload.kafka_input_partitions:
return "Kafka Streams"
return "Kafka Streams plus repartitioning, or a cluster engine"
if workload.latency_budget_ms < 1000:
if workload.large_keyed_state or workload.complex_event_time:
return "Apache Flink"
if workload.kafka_to_kafka_stateless:
return "Kafka Streams or Flink"
if workload.team_standardizes_on_spark_sql:
if workload.latency_budget_ms >= 1000:
return "Spark Structured Streaming"
if workload.batch_backfills_must_share_code_with_live_streams:
return "Beam, Flink, or Spark depending on runtime ownership"
return "Prototype Flink and Spark; reject by sink semantics and ops cost"Flink’s record-at-a-time execution is a strong fit for sub-second event-driven applications. Spark’s micro-batch mode amortizes scheduling and optimization overhead across batches; Spark’s own documentation describes micro-batch latency as low as roughly 100 ms with exactly-once guarantees and continuous processing as roughly 1 ms with at-least-once guarantees for supported queries. Kafka Streams can be low-latency because it runs in-process and processes records from local buffers, but rebalances and state restoration can dominate tail latency during failures.
Flink checkpoints operator state and source positions together. Spark records offsets, state-store data, and batch progress in checkpoint locations. Kafka Streams writes state updates to local stores and internal changelog topics; if a task moves, the state may be rebuilt by replaying the changelog unless a standby replica is caught up. Beam’s state behavior is only meaningful after choosing the runner.
Exactly-once is not a single feature switch. It requires replayable input, deterministic computation, checkpointed state, and a sink that can commit idempotently or transactionally. Flink’s checkpoint barriers give exactly-once state by default. Spark micro-batch can produce exactly-once results with replayable sources and idempotent or transactional sinks. Kafka Streams exactly-once semantics depend on Kafka transactions and state/changelog consistency. Beam inherits the runner and sink guarantees.
result_key = {
"pipeline": "orders-hourly-v3",
"window_start": window_start.isoformat(),
"window_end": window_end.isoformat(),
"merchant_id": merchant_id,
}
def write_window_result(db, key, amount_cents, source_offsets):
db.execute("""
INSERT INTO hourly_revenue
(pipeline, window_start, window_end, merchant_id, amount_cents, offsets_json)
VALUES
(:pipeline, :window_start, :window_end, :merchant_id, :amount_cents, :offsets_json)
ON CONFLICT (pipeline, window_start, window_end, merchant_id)
DO UPDATE SET
amount_cents = EXCLUDED.amount_cents,
offsets_json = EXCLUDED.offsets_json
""", key | {
"amount_cents": amount_cents,
"offsets_json": json.dumps(source_offsets),
})These workloads often need sub-second reaction, event-time correctness, keyed state, timers, and reliable recovery. Flink is usually the first engine to evaluate. Kafka Streams can fit when the logic is embedded in a Kafka-centric service and state is partition-aligned. Spark micro-batch is usually less natural unless the latency budget is loose and the team already operates Spark heavily.
When the output is hourly aggregates, incremental tables, and data lake updates, Spark Structured Streaming is attractive because it shares DataFrame and SQL tooling with batch ETL. Beam is attractive when the same pipeline needs to run on managed Dataflow or switch runners. Flink is still a strong option when late data, sessionization, and continuous updates matter more than SQL ecosystem consolidation.
Kafka Streams is designed for applications that want stream processing without deploying a separate processing cluster. A service can consume topics, maintain local state, join streams, and write results back to Kafka. This is a good fit for domain-owned transformations, materialized views, and routing logic where Kafka is already the operational backbone.
Beam is useful when the organization wants one semantic model for batch and streaming while retaining runner choice. The trade-off is that portability must be tested. Advanced state, timers, side inputs, and connectors can have runner-specific limitations, so a production Beam decision should include a compatibility matrix, not just the claim that the API is portable.
A stream engine comparison should include worker crashes, source lag, sink errors, partition rebalances, late events, and backfills. The selected engine must preserve the contract when a checkpoint fails or a stateful task migrates. Kafka Streams documentation notes that restoring state by replaying changelog topics can dominate task initialization time; that cost is invisible in a small happy-path benchmark.
Every derived output needs a contract: append-only, upsert, retract, or compensate. Flink and Beam pipelines may emit early and late panes. Spark update mode writes changed rows per micro-batch. Kafka Streams materialized views update continuously. Downstream teams should know whether a number is final, whether it can decrease, and how corrections are represented.
Ten thousand test keys do not reveal the cost of a hundred million production keys. Include hot keys, skewed partitions, large windows, and realistic late-arrival distributions. Measure checkpoint size, restore time, RocksDB or state-store disk usage, output latency, and sink commit time.
load_test_plan = {
"input_rate_per_sec": [10_000, 50_000, 100_000],
"active_keys": [1_000_000, 20_000_000],
"hot_key_fraction": 0.02,
"late_event_p99_seconds": 900,
"failure_drills": [
"kill_worker",
"pause_one_source_partition",
"restart_job_manager",
"make_sink_return_retriable_errors",
"force_kafka_streams_rebalance"
],
"acceptance": {
"p95_output_latency_ms": 2000,
"zero_lost_records": True,
"duplicate_external_effects": 0,
"restore_time_minutes": 10
}
}Stateful joins and aggregations require related records to meet at the same logical task. In Kafka Streams this usually means co-partitioned topics and stable keys. In Flink or Spark, repartitioning is possible but creates shuffle cost and operational pressure. A poor key can turn any engine into a hot-spot generator.