A vendor-neutral reference architecture for the modern data platform: the functional layers every team needs - ingestion, storage, processing, serving, orchestration, governance, observability - and the real open and managed options at each one.
Most "modern data platform" diagrams are sales tools. A vendor draws seven boxes, and surprise: every box maps to a product they sell. That is a pricing sheet with arrows.
This post takes the opposite stance. A data platform is a set of functional layers, not a list of products. If you understand the layers and what each one has to do, you can assemble a platform from open-source components, managed services, or a mix, and swap any single piece without rebuilding the rest. What follows is that layer map and the real options at each layer, written for the engineer who has to operate it.
This is the 2026 view. We published a prior-generation architecture map in 2022, and the stack has shifted since: streaming is the default ingestion mode in the platforms we build, open table formats won the storage argument, and governance plus observability moved from "nice to have" to mandatory at scale. We reference the older piece as a companion rather than repeat it.
The seven functional layers
Strip away brand names and every credible architecture exposes the same seven responsibilities: ingestion and change capture, storage and table format, processing and transformation, serving and query, orchestration, governance, and observability.
The layers are responsibilities, not servers. One tool can cover several (ClickHouse is both storage and a serving query engine) and one layer can be split across several tools. Thinking in layers forces you to ask "is this responsibility covered?" before "which product?". Vendor diagrams usually collapse this into five boxes (ingestion, storage, processing, consumption, governance). For engineers who carry the pager, orchestration and observability deserve to stand on their own, because they are where platforms quietly fail.
Read the table as "for each row, pick at least one tool; you do not need every column."
| Layer | What it has to do | Representative open / self-hosted | Representative managed |
|---|---|---|---|
| Ingestion + CDC | Get data in, capture row-level changes from source databases | Debezium, Kafka Connect, Airbyte, Apache NiFi | Fivetran, AWS DMS, Confluent Cloud, Estuary |
| Storage + table format | Durable, queryable storage with schema, snapshots, time travel | Apache Iceberg, Delta Lake, Apache Hudi on object storage | Snowflake, Databricks, BigQuery, S3 Tables |
| Processing + transformation | Reshape, join, and model raw data into trusted tables | Spark, Flink, dbt Core, SQLMesh | dbt Cloud, Databricks, Confluent Flink, EMR |
| Serving + query | Answer analytical and operational queries with low latency | ClickHouse, Trino, DuckDB, Apache Druid | ClickHouse Cloud, Athena, BigQuery, Snowflake |
| Orchestration | Schedule, sequence, retry, and backfill pipelines | Apache Airflow, Dagster, Prefect | Astronomer, Dagster+, MWAA, Google Cloud Composer |
| Governance | Catalog, lineage, access control, classification | DataHub, OpenMetadata, Apache Polaris, Unity Catalog OSS | Unity Catalog, Snowflake Horizon, Atlan, Collibra |
| Observability | Detect freshness, volume, and quality regressions before users do | OpenLineage, Great Expectations, Elementary | Monte Carlo, Metaplane, Bigeye |
A tool appearing in two rows is not a mistake. Storage engines grew query layers, query engines learned to write open table formats, and catalogs absorbed lineage and observability. The layer model tells you what a converged tool is actually doing under the hood.
Storage and the table format question
Storage is where the 2022-to-2026 shift is most visible. An open table format (Apache Iceberg, Delta Lake, Apache Hudi) turns a directory of Parquet files in object storage into a real table, with schema evolution, ACID transactions, and time travel, readable by any compliant engine.
What changed is that the major warehouses stopped fighting it. Databricks previewed full Apache Iceberg support through Unity Catalog's Iceberg REST Catalog API in June 2025 and took Managed Iceberg, Iceberg v3, and Foreign Iceberg GA in May 2026, meaning any REST-spec client (Spark, Flink, Trino) can read and write the same managed tables. Snowflake exposes the same surface through its Iceberg tables and the Polaris catalog, and Polaris itself became an Apache top-level project in February 2026. The practical result: your table format is now a more durable decision than your query engine. Pick the format deliberately, and engines become swappable clients against it. Concretely: swapping Trino for ClickHouse on the hot path over the same Iceberg tables is a new connection and a set of materialized projections, not a data copy, and the dbt models that produced the tables do not change. If you are weighing the two front-runners, we cover the trade-offs in Apache Iceberg vs Delta Lake.
The other half of storage is the hot path. A lakehouse on Iceberg is excellent for large historical scans and cheap retention, but a Parquet-on-S3 query that takes two seconds is a non-starter behind a user-facing dashboard. That is where a columnar serving engine earns its place. The common pattern is two tiers: open table format for the durable, governed, everything-lives-here layer, and a fast columnar store (ClickHouse, Druid) for the sub-second slice that powers product analytics and real-time dashboards. The lakehouse is the source of truth; the serving store is a materialized projection of what needs to be fast. If you are still deciding between warehouse, lake, and lakehouse for this layer, see EDW vs data lake vs lakehouse, what is a data lakehouse, and the medallion architecture.
Ingestion: streaming is the default, batch is the special case
The biggest mental shift since the last generation is that streaming stopped being the exotic option. Batch is just a bounded stream you run on a schedule. Most new pipelines we see start from a continuous event log and add batching only where latency does not matter.
Three ingestion patterns cover almost everything, and most real platforms run all three:
- Change data capture (CDC). Stream row-level inserts, updates, and deletes out of operational databases by reading the transaction log. Debezium into Kafka is the open-source standard; AWS DMS and Fivetran are the managed equivalents. CDC is how you get OLTP data into the analytics plane without hammering production with polling queries, a split we examine in OLTP vs OLAP in 2026.
- Event streaming. Application and clickstream events land directly on Kafka or a Kafka-compatible log (Redpanda, Confluent Cloud), then fan out to storage and stream processors.
- Batch and zero-ETL. Periodic loads for legacy systems and slow-moving reference data, plus the managed "zero-ETL" replication paths cloud vendors offer between their own services.
The pitfall is treating CDC as fire-and-forget. CDC changelogs are not append-only event streams; a downstream consumer has to interpret update and delete semantics or it will double-count. As Gunnar Morling shows in his walkthrough of ingesting Debezium events with Flink SQL, the connector and format together decide whether events are read as an append-only stream or a changelog, and getting that wrong corrupts every aggregate downstream. Design the ingestion contract before you pick the tool. Our primer on what a data pipeline is sets the foundations, and ETL process optimization covers tuning the batch paths you keep.
Processing, serving, and orchestration
Processing and transformation is dominated by declarative SQL modeling. dbt established the pattern (version-controlled SQL models, tests, generated docs) and SQLMesh extended it with column-level lineage and cheaper environment management. The consolidation became literal: Fivetran and dbt Labs announced an all-stock merger in October 2025 and closed it on June 1, 2026, and Fivetran's earlier acquisition of Tobiko Data ended with SQLMesh donated to the Linux Foundation in March 2026. dbt Core, now on the Rust-based v2 engine shared with Fusion, remains Apache-2.0. The same dbt-style workflow now reaches streaming too: Confluent's dbt adapter for Flink SQL lets you define streaming tables and sources over Kafka topics as dbt models, so skills built on Snowflake or BigQuery transfer to the real-time side instead of living in a separate Terraform-and-console world. For a concrete streaming-first stack, see our Kafka, Flink, and ClickHouse blueprint.
Serving and query is the layer your users touch. The split is between interactive analytics over the lakehouse (Trino, Athena, DuckDB for local and embedded work) and low-latency serving from a columnar engine (ClickHouse, Druid). If a human is waiting on the query inside an application, you want a purpose-built columnar store, not a lakehouse scan.
Orchestration ties it together. The question an orchestrator answers is "what runs after what, what happens when it fails, and how do I backfill three months of history without a war room." Airflow remains the incumbent; Dagster and Prefect compete on asset-centric and Pythonic developer experience. Pick based on whether your team thinks in tasks (Airflow) or in data assets (Dagster). Either way, this layer stops being optional the moment you have more than a handful of dependent jobs.
Governance and observability
These two layers separate a platform from a pile of pipelines, and they are where most teams under-invest until an incident forces the issue.
Governance is the catalog, lineage, access control, and classification that let people find data and trust it. The open standard to build around is OpenLineage, which defines how jobs across Airflow, Spark, Flink, and dbt emit structured lineage events as they run. Catalogs like DataHub and OpenMetadata consume those events and build navigable lineage graphs. New in 2026: catalogs are becoming the context layer for AI agents. DataHub exposes its metadata graph over the Model Context Protocol, so AI tools can read governed context and write back tags, descriptions, and ownership. Agents read the catalog the same way analysts do, which is a new reason to keep it accurate.
Observability is monitoring for the data itself: freshness, volume, schema drift, quality, detected before a stakeholder notices a dashboard went stale. Open tools (Great Expectations, Elementary, the OpenLineage spine) and managed platforms (Monte Carlo, Metaplane) cover the same job: catch it before the CFO's dashboard goes stale. Treating pipelines with the same operational rigor as production software is what DataOps is about. Instrument lineage and quality from day one. Retrofitting observability onto a platform that has already lost its users' trust costs far more than building it in.
Right-sizing: what to skip
The full seven-layer architecture is what a platform looks like at scale. It is not what a five-person data team should build on day one. The most common failure we see is a startup assembling an enterprise stack (a separate tool per layer, a dedicated catalog, a streaming bus) for a workload a single Postgres replica and dbt could serve.
A reasonable progression:
- Early stage (one data engineer, sub-terabyte, a handful of analysts). Managed warehouse or a single columnar engine, dbt for transformation, scheduled loads. Skip the streaming bus, the standalone catalog, and the dedicated observability vendor. Lineage from dbt and warehouse-native access controls are enough.
- Growth stage (a platform team, multi-terabyte, real-time needs emerging). Introduce an open table format so you are not locked to one engine, add CDC and a streaming path for the workloads that need it, and stand up a real catalog and observability layer. A typical stage-2 stack: Debezium into Kafka, Flink or Spark writing Iceberg on S3, dbt on top, ClickHouse for the hot path, Airflow, DataHub.
- Enterprise (central platform plus embedded teams, petabyte-scale, regulatory exposure). All seven layers, deliberately. Governance and observability become mandatory here, and regulation is heading the same way: the EU AI Act's high-risk data-governance obligations (Article 10), now due December 2027 after the 2026 Digital Omnibus deferral, require that training and input data be documented and auditable.
The layer model helps because it tells you what you are skipping: a responsibility you have decided you do not yet need, with a name, so you know what to add when the time comes.
Building or modernizing a platform like this? Architecture and implementation for these stacks is what our data engineering consulting practice does, from streaming ingestion through the analytics layer.
