A data engineering view of edge AI for real-time analytics: where inference runs, how edge events stream back to a central OLAP store like ClickHouse, and the latency and consistency trade-offs that decide the design.
Most writing on edge AI stops at "run the model on the device." That framing misses the harder half of the problem. The model is the easy part. The real engineering work is the data plane around it: where features come from, where inference results go, how thousands of remote sites stay consistent with a central store, and how you query the whole fleet fast enough to act on it. Edge AI for real-time analytics is a data engineering problem first and a model problem second.
Two Flavors of "Real-Time at the Edge"
There are two distinct things people mean by edge AI, and most production systems need both.
Inference-at-edge runs a model on or near the device that generates the data: a camera, a PLC on a factory line, a vehicle gateway, a point-of-sale terminal. The decision is made locally, in single-digit to low-double-digit milliseconds, without a network hop. This is what keeps a control loop safe or a checkout flowing when the uplink is congested or down.
Streaming-back-to-center treats every edge event (sensor readings, inference outputs, confidence scores) as a record to ship into a central analytics backend. That backend is where you ask cross-fleet questions: which sites are drifting, what changed after the last model rollout, how does this hour compare to the same hour last week.
Inference-at-edge optimizes for the latency of a single local decision. Streaming-back-to-center optimizes for queryable history across the whole fleet. A serious deployment runs both at once: the edge decides, the center learns.
The two paths have opposite design pressures. The edge path wants minimal dependencies, bounded timing, and graceful degradation under network loss. The central path wants completeness, schema discipline, and fast ad-hoc query. Routing every decision through the cloud so it lands in your warehouse conflates the two, and produces a system that is neither fast nor reliable.
A Reference Architecture, End to End
A workable pattern moves data through four stages:
Edge devices (sensors, cameras, PLCs)
-> Local inference (ONNX Runtime / TensorRT / TFLite on a gateway)
-> Edge gateway: normalize protocols (OPC-UA, Modbus), pre-aggregate, buffer
-> Event transport (MQTT broker -> Kafka)
-> Central real-time OLAP (ClickHouse / Pinot / Druid)
-> Dashboards, alerting, anomaly detection, retraining triggers
The gateway is the load-bearing component. Industrial protocols are messy and per-vendor; the gateway normalizes them into a common event schema before anything leaves the site. EMQ's production reference for this exact shape (its NeuronEX gateway normalizing OPC-UA and Modbus, MQTT into EMQX, through Kafka, into ClickHouse Cloud) delivers sub-second analytics even across public networks. EMQ separately reports its hosted EMQX cloud reaching 2 million messages per second on a 23-node cluster (ClickHouse blog, 2025).
On the ClickHouse side, ingestion follows the three-layer pattern from the Kafka engine docs: a Kafka engine table consumes the stream, a materialized view transforms each batch, and a ReplicatedMergeTree table stores it for query. Roughly:
-- 1. Source: read raw edge events off Kafka
CREATE TABLE edge_events_queue (
site_id LowCardinality(String),
device_id String,
ts DateTime64(3),
metric Float64,
inference String,
confidence Float32
) ENGINE = Kafka SETTINGS
kafka_broker_list = 'kafka:9092',
kafka_topic_list = 'edge.events',
kafka_group_name = 'clickhouse-edge-events',
kafka_format = 'JSONEachRow';
-- 2. Destination: partitioned, sorted for fleet queries
CREATE TABLE edge_events (
site_id LowCardinality(String), device_id String,
ts DateTime64(3), metric Float64,
inference String, confidence Float32
) ENGINE = ReplicatedMergeTree
ORDER BY (site_id, device_id, ts)
PARTITION BY toYYYYMMDD(ts);
-- 3. Materialized view wires the queue to the table
CREATE MATERIALIZED VIEW edge_events_mv TO edge_events
AS SELECT * FROM edge_events_queue;
ReplicatedMergeTree with no arguments works on ClickHouse Cloud and on self-hosted clusters where default_replica_path and default_replica_name are configured; otherwise pass the ZooKeeper path and replica name explicitly (replication docs).
Why a columnar OLAP store and not a transactional database or a search engine for this tier? Because the central question is analytical: scans over hundreds of millions of edge events, grouped by site and time, with sub-second response. That is the workload ClickHouse, Pinot, and Druid are built for. We default to ClickHouse for this; Pinot and Druid earn a look when you already run them or need their upsert and real-time segment model at very high ingest concurrency. For a broader walk through tool selection at this layer, see our engineer's guide to real-time data analysis tools, and for the streaming side specifically, the Kafka, Flink, and ClickHouse architecture blueprint.
Latency Budgets: When the Edge Actually Pays Off
Edge compute is not free. It adds fleet management, hardware heterogeneity, and a second deployment target. You only take that on when the latency math demands it, so do the math before committing.
In the systems we have built, local inference on a gateway CPU or NPU lands in single-digit to low tens of milliseconds. A full cloud round-trip (sensor data up, queue, inference, command back down) runs 50-500 ms and varies badly under congestion; for a device on another continent from a single central region, a few hundred milliseconds is normal. More important than the average is the variance. Removing the network from the decision loop makes latency bounded, which is what a control loop needs.
Measure your own budget before deciding: instrument the round-trip from event to actuation at p50 and p99 over a week, including the bad network days. The p99 is the number that decides the architecture.
| Dimension | Inference at the edge | Inference in the cloud |
|---|---|---|
| Decision latency | ~1-20 ms, bounded | 50-500 ms, network-dependent |
| Behavior under network loss | Keeps working locally | Stalls or fails |
| Compute available | Constrained (CPU/NPU, limited RAM) | Effectively unlimited |
| Model size ceiling | Tight; needs quantization/pruning | Large models run fine |
| Cross-fleet visibility | None on its own | Native |
| Operational cost | Fleet mgmt, OTA, heterogeneity | Higher per-inference and egress |
If a missed deadline causes physical harm or lost transactions (a safety interlock, a collision-avoidance loop, a checkout that must not block), inference belongs at the edge. If the decision tolerates a few hundred milliseconds and benefits from a larger model or richer context, keep it central and stream the events back. A retail loss-prevention camera flags a suspect event locally in milliseconds; the same event still streams to ClickHouse so the fleet can be queried for patterns. Most real deployments draw this line per decision, not per system.
Data-Plane Patterns: What to Send and What to Keep
The link between thousands of sites and the center is the scarce resource. How you treat it shapes both cost and forensic capability.
- Pre-aggregation at the edge. Roll up high-frequency telemetry into per-minute or per-event summaries before transmitting. A vibration sensor sampling at kHz does not need every raw point in the center; p95, peak, and a drift indicator usually carry the signal. This is the biggest single lever on bandwidth and central storage.
- Raw passthrough for forensics. Keep full-fidelity raw data when you need to reconstruct exactly what the model saw, for incident review, audit, or retraining. Buffer it at the edge and ship on demand rather than continuously.
- Selective sampling. Always send anomalies and low-confidence inferences at full detail; sample the high-confidence majority. Inference confidence is a natural sampling key: the cases the model is unsure about are the ones worth keeping.
- Tiered hot/cold storage. In the central store, keep recent edge events on fast storage for sub-second query, and age older partitions to object storage. ClickHouse tiered storage and TTL moves, or an Iceberg lakehouse for the cold tier, give you queryable history without paying hot-tier prices for last quarter's telemetry.
A common mistake is choosing one globally. Pre-aggregating everything destroys the forensic trail; passing everything through raw blows the budget. Pick per data stream, driven by how that stream gets used.
Model Lifecycle and Feedback Across Thousands of Sites
A model trained once and forgotten is a liability. Edge environments drift: lighting changes, machines wear, traffic patterns shift, and the model degrades silently unless you instrument it. Managing this across a large fleet is where edge AI projects most often fail operationally.
Treat rollout like any other risky deploy. Push new model versions over the air with delta packaging to limit bandwidth, roll out to a canary subset first, watch accuracy and resource metrics, and keep a tested rollback path before going fleet-wide. Version every artifact and stamp the model version onto every inference event. That last detail is what makes the central store useful: when you can GROUP BY model_version over millions of edge events, a regression after a rollout shows up as a measurable shift, not a support ticket three weeks later.
The feedback loop closes through the same pipe. Edge inference results (prediction, confidence, and where available the eventual ground-truth outcome) stream back into the central OLAP store. There you compute drift: compare the live distribution of confidence and predictions against the training baseline, per site and per model version, and trigger retraining when the gap crosses a threshold. The central store does double duty here. It holds the queryable history of edge events and it is the observability plane for the fleet itself: which devices are offline, which are slow, where inference quality is sliding (more on this in why databases are critical in the agentic AI era).
A query to surface drift candidates is ordinary OLAP work:
SELECT site_id, model_version,
avg(confidence) AS mean_conf,
countIf(confidence < 0.6) / count() AS low_conf_ratio
FROM edge_events
WHERE ts >= now() - INTERVAL 1 HOUR
GROUP BY site_id, model_version
HAVING low_conf_ratio > 0.2
ORDER BY low_conf_ratio DESC;
Pitfalls That Sink Edge Deployments
- Model size versus hardware. Edge accelerators have hard memory and compute ceilings. Plan for quantization and pruning from the start; do not design around a model the target hardware cannot run.
- Hardware heterogeneity. A fleet built over years spans CPU generations, NPUs, and OS versions. The deployment system has to handle multiple target runtimes (ONNX Runtime, TensorRT, TFLite) rather than assuming one.
- Network partitions. The edge will lose its uplink. Local inference must keep deciding, and the gateway must buffer events durably and reconcile when the link returns. Design for the partition; it is not an edge case, it is Tuesday.
- Consistency between edge and center. The edge holds the recent, locally-true state; the center holds the complete, eventually-consistent picture. You cannot have both strong consistency and edge autonomy under partition. Pick eventual consistency deliberately, stamp events with edge-generated timestamps and idempotency keys, and let the central store deduplicate.
That last trade-off is the heart of the architecture. Choosing edge autonomy means accepting that the central view lags reality by the buffer-and-ship interval, and designing every downstream query and alert to tolerate it.
Get the data plane right and the model becomes a swappable part. Get it wrong and no model will save the project. If you are designing an edge-to-center analytics platform and want a second pair of eyes on the architecture, talk to us; this is the kind of system we build and operate.
