Batch, micro-batch, streaming and CDC are cost profiles, not camps. Where pipeline spend actually lands, a worked CDC to Kafka to Flink to ClickHouse and Iceberg pipeline, a decision table and a checklist for picking a pattern per data flow.
Every pipeline cost review we do ends at the same two line items. The first is transformation compute: the warehouse or cluster hours spent recomputing things that did not change. The second is always-on infrastructure serving freshness that no consumer reads. Neither has much to do with which tool you picked. Both come from applying the wrong pattern to a data flow, usually by defaulting to whatever the last project used.
So the batch-versus-streaming question is a cost question, and it has to be answered per flow rather than per platform. A mature platform runs a nightly batch, a Kafka-fed live view and a CDC replica side by side, and that is fine. What is not fine is a nightly report fed by a 24/7 Flink job, or a "real-time" dashboard fed by a connector that polls every five minutes. This post is about telling those apart before you build them.
Where the money goes
Transformation is the expensive half. Moving bytes from A to B is cheap on every cloud; recomputing a table from scratch is where warehouse bills come from. Snowflake makes the arithmetic visible: an X-Small warehouse bills 1 credit per hour, a Small 2, a Medium 4, a Large 8, an X-Large 16, and each size step approximately doubles both compute and credits (Snowflake warehouse overview). A dbt job that ran on a Small and got bumped to an X-Large "to be safe" now pays 8x for the same query plan. Billing is per second with a 60-second minimum on every start or resume (Snowflake compute costs), so a warehouse that auto-suspends and wakes for a dozen tiny models an hour also pays a minute each time.
The fix is older than any of the tools. dbt's incremental materialization transforms all rows on the first run and only the rows you filter for on later runs, which the dbt docs describe as "vastly reducing the runtime of your transformations" and reducing compute cost (dbt incremental models). A full-refresh model over three years of events, scheduled hourly, is the single most common thing we delete on an engagement.
The second driver is duplicated logic. Jay Kreps' 2014 critique of the Lambda architecture still holds: "maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems like it would be" (Kreps, O'Reilly Radar). His alternative, reprocess by replaying the retained log through a second instance of the same stream job, is what people now call Kappa. In the field the shape that won is neither pure Lambda nor pure Kappa: one streaming or micro-batch path for live views, and a lakehouse table format holding full history, with transforms living in exactly one place.
The four patterns and what decides between them
Definitions in one paragraph, then a table. Batch reads a bounded window on a schedule and pays only while the job runs. Streaming consumes a log continuously with a stateful processor such as Flink, and pays for a process that never stops plus checkpointing and exactly-once sinks. Micro-batch groups events arriving within a short trigger interval into one small job; the Spark docs put its end-to-end latency "as low as 100 milliseconds" with exactly-once guarantees via checkpointing and write-ahead logs, and its Continuous Processing mode at around 1 ms with at-least-once (Spark Structured Streaming). CDC reads the database transaction log and emits only changed rows, so steady-state work tracks the change rate rather than the table size.
| Pattern | Pick it when | Latency you actually get | What you pay for |
|---|---|---|---|
| Batch | Consumers read on a schedule (reports, ML training sets, month-end) | Minutes to hours, by design | Compute only during the job; cheapest per row |
| Micro-batch | A dashboard or feed needs "fresh enough" and you already run Spark | Seconds to minutes; 100 ms floor per Spark docs | Always-on driver plus amortized batches |
| Streaming | Per-event decisions: fraud, alerting, feature serving | Milliseconds to seconds | 24/7 stateful cluster, checkpoints, exactly-once sinks, on-call |
| CDC | Source is an OLTP database and the target must mirror it | Seconds with Debezium on Kafka; minutes or more with DMS, which offers no latency SLA (AWS DMS) | Change volume, plus an initial snapshot and log retention on the source |
Two things the table hides. CDC's "proportional to change volume" holds only in steady state: the initial snapshot is a full read of every captured table, and if the connector stops for long enough the source keeps write-ahead log segments around for it, which costs disk on the database (Debezium PostgreSQL connector). And streaming's price is mostly operational. Flink checkpoints require a replayable source and durable state storage (Flink checkpointing); exactly-once into Kafka means every write goes inside a Kafka transaction committed on checkpoint (Flink Kafka connector). Someone has to understand that at 3am.
One pipeline, end to end
Take an orders table in PostgreSQL. Finance wants a daily revenue rollup, ops wants a live order-status board, and data science wants three years of history to replay. Three consumers, three freshness needs, one source. Here is how the patterns compose:
- Capture once. Debezium takes a consistent snapshot of
orders, records the WAL position, then streams committed changes from that position through a logical replication slot into a Kafka topic. Monitor the slot: an idle connector pins WAL on the primary. - Live view via streaming. A Flink job consumes the topic, keys by
order_id, keeps the latest status per order, and writes to ClickHouse. Batch the sink: ClickHouse wants inserts of 10,000 to 100,000 rows because every insert creates a part that later has to be merged, and async inserts move that batching server-side if the client cannot do it (ClickHouse insert strategy). The ops board reads ClickHouse. Latency is a few seconds, dominated by the insert batch window. - History via the lakehouse. The same Flink job, or a second one reading the same topic, appends raw change events to an Iceberg table. The Iceberg sink commits in Flink's
notifyCheckpointCompletecallback, so the checkpoint interval is the commit interval, and a 10-second checkpoint means 8,640 snapshots a day of small files untilrewriteDataFilescompacts them (Iceberg Flink writes). Pick a checkpoint interval in minutes for this sink; the live path already covers seconds. - Rollups via batch. A nightly dbt incremental model reads yesterday's partition from Iceberg (or the warehouse it is registered in) and updates the revenue table. Data science replays history with
SELECT ... FROM orders_history TIMESTAMP AS OF '...'or an incremental read between two snapshot IDs (Iceberg time travel), no second copy of the transformation code required.
The revenue logic exists once, in dbt. The status logic exists once, in Flink. Nothing is computed twice, and the only always-on components are the ones serving the consumer that reads continuously. Our Kafka, Flink and ClickHouse blueprint goes deeper on steps 2 and 3, and the Debezium production patterns post covers what goes wrong in step 1.
Anti-patterns from cost reviews
Hidden batch inside a "real-time" pipeline is the most common. A dashboard advertised as live, fed by a JDBC connector on a five-minute poll, is a micro-batch pipeline with a streaming label, and someone downstream has promised an SLA the slowest hop cannot meet. Measure end-to-end latency at the consumer before you put a number in a contract.
Streaming for slow-moving data is the most expensive. A dimension table that changes twice a day does not need Kafka, a stream processor, a schema registry and a pager rotation. A scheduled sync is simpler, and simpler is cheaper every month after launch.
Over-orchestration is the most annoying. One SQL transform wrapped in a six-task DAG with sensors, retries and branching adds surface area and no value. Orchestration earns its place when there are real cross-system dependencies.
Duplicated transforms are the most dangerous. The same business rule written once in Spark for the nightly path and again in Flink for the live path will drift, and when finance and ops disagree about yesterday's revenue, both pipelines are correct according to their own code.
The decision checklist
Ask these four questions per data flow, in order. The first one that gives a clear answer usually decides.
- How often does anyone read the output? Once a day: batch. Every few minutes: micro-batch. Per event, with a decision attached: streaming. Get the answer from whoever reads the output; producers always say "as fresh as possible".
- Does the source expose a change log? An OLTP database with WAL or binlog access: CDC, and the freshness downstream is whatever step 1 needs. A SaaS API or a file drop: batch or micro-batch, because there is nothing to tail.
- What is the slowest hop? Find it before promising latency. If it is a five-minute poll, everything after it is micro-batch no matter what runs there.
- Will you need to reprocess history? If yes, land raw events in a table format with snapshots and time travel, and keep the transform in one engine so replay uses the same code as production.
If you are choosing between these patterns for a specific workload, or trying to untangle a platform that grew three of them by accident, our data engineering team does this work. The ETL optimization field guide covers the diagnosis side, and the modern data platform guide covers the stack around it.
