Itamar Syn-Hershko
ClickHouse observability means watching the background machinery - merges, TTL, replication, Keeper - not just query latency. The silent failure modes of ClickHouse, the system-table queries that surface them early, and where AI-assisted maintenance fits.
The worst ClickHouse incidents don't start with an alert. A merge backlog builds for days, a materialized view quietly drops rows, a replica serves stale reads - and every dashboard stays green until ingestion halts or someone notices the numbers don't add up.
ClickHouse observability is the practice of tracking the database's deferred background work - merges, mutations, TTL cleanup, replication, and Keeper coordination - rather than only the query path. The distinction matters because ClickHouse is fast precisely because it defers work, and deferred work fails silently. This post walks through the failure modes that hide from standard dashboards, the system tables that expose them, and what to do once you can see them. If you're choosing a monitoring stack first, start with our comparison of ClickHouse monitoring tools and come back here for what to actually watch.
Why ClickHouse Fails Quietly
Most databases do their expensive work at write time. ClickHouse inverts this: an INSERT just writes an immutable part to disk, and everything else - merging parts, applying TTL deletes, replicating data, materializing mutations - happens later, in background pools, eventually. That design is why a modest cluster ingests millions of rows per second.
It's also why the failure surface looks nothing like a typical OLTP database. When background machinery stalls, foreground operations keep succeeding. Inserts still return OK while parts pile up. Queries still answer while a replica drifts behind. Disk usage creeps while TTL cleanup silently stops. There is no error to page on, because from ClickHouse's perspective nothing has failed yet - work is just pending. The gap between "pending" and "outage" is where observability earns its keep.
ClickHouse is honest about all of this: it exposes over a thousand metrics through system tables like system.parts, system.merges, system.replicas, and system.mutations. The problem is rarely missing data. It's knowing which handful of signals predict outages, and checking them continuously.
Six Silent Failure Modes and Their Early Signals
Each of these comes from real production incidents we've worked on across ClickHouse consulting engagements. None of them shows up on a CPU/memory dashboard until late in the game.
| Failure mode | Why it's silent | Early signal |
|---|---|---|
| Merge backlog | Inserts are throttled first, and still succeed | MaxPartCountForPartition trending toward 1000 |
| Materialized view drops data | Source insert succeeds; MV write is not atomic with it | Row-count drift between source and target tables |
| TTL stops keeping up | TTL runs only inside merges, at low priority | Disk growth with retention unchanged |
| Replica divergence | Stale replicas keep serving reads by default | absolute_delay in system.replicas |
| Detached parts accumulate | Invisible to table-level metrics, never auto-cleaned | Rows in system.detached_parts |
| Shard skew | Cluster-wide averages mask the hot shard | Per-shard part counts and insert rates diverging |
Merge backlog. Parts accumulate faster than the merge pool clears them. ClickHouse delays inserts at 1,000 parts per partition and rejects them at 3,000 with TOO_MANY_PARTS - and because the delay phase is invisible to callers, the first symptom anyone notices is often the outage itself. Watch parts per partition directly:
SELECT database, table, partition, count() AS parts
FROM system.parts
WHERE active
GROUP BY database, table, partition
ORDER BY parts DESC
LIMIT 10;
We cover causes and fixes in depth in the too many parts problem.
Materialized views dropping data. An insert into a source table and the writes into its materialized views are not atomic. When an MV write fails mid-insert, the source table keeps the rows and the target quietly doesn't - every downstream rollup under-counts, and the hole surfaces weeks later during a reconciliation. Periodic row-count checks between source and MV targets are cheap insurance; see our materialized views guide for how cascades behave under failure.
TTL falling behind. TTL deletes only happen inside merges, and TTL-driven merges are rate-limited (merge_with_ttl_timeout defaults to four hours per table) and deprioritized when the pool is busy. Retention quietly stops working, and disk usage climbs until a replica goes read-only. Track free space per volume in system.disks alongside the merge queue.
Replica divergence. A Keeper hiccup leaves a replica stale, and by default distributed queries keep reading from replicas that lag by up to 300 seconds (max_replica_delay_for_distributed_queries). The same query returns different numbers depending on which replica answers. Meanwhile a replica that loses its Keeper session altogether goes read-only:
SELECT database, table, is_readonly, absolute_delay, queue_size
FROM system.replicas
WHERE is_readonly OR absolute_delay > 30;
More on this in ClickHouse replication health.
Detached parts. Broken and unexpected parts land in the detached folder, where ClickHouse never cleans them up. They're invisible to every table-level metric until the disk fills or a restart chokes on them. system.detached_parts with a reason breakdown belongs on a weekly review, if not an alert.
Shard skew. A skewed sharding key piles writes onto one shard. Cluster averages look fine while that shard hits TOO_MANY_PARTS first - and ClickHouse has no automatic rebalancing to save you. Always chart per-shard, never just per-cluster.
Monitoring, Observability, and Maintenance Are Different Jobs
These terms get used interchangeably, and the sloppiness has a real cost: teams buy a monitoring stack and believe they've bought the other two.
| Layer | Question it answers | Typical tooling |
|---|---|---|
| Monitoring | Is a metric outside its threshold right now? | Grafana, Datadog, the built-in /dashboard |
| Observability | Can I explain why from the signals I collect? | System tables, query_log analysis, per-shard views |
| Maintenance | Is someone acting on it before it becomes an incident? | Runbooks, expert review, AI-assisted platforms |
Monitoring tells you a partition crossed 800 parts. Observability connects that to the single-row inserts from a new service that bypassed your batching layer. Maintenance is enabling async_insert for that client, or fixing the batch size at the source - before the 1,000-part throttle kicks in. Our ClickHouse monitoring tools comparison covers the first layer well; the queries above get you much of the second. The third has traditionally meant having a ClickHouse expert on staff, which is exactly the scarce resource most teams are missing.
Closing the Loop with AI-Assisted Maintenance
The maintenance layer is the one we kept getting pulled into as consultants, and it's why our team built NeverBlink - which now supports ClickHouse alongside Elasticsearch and OpenSearch, covering ClickHouse Cloud, self-hosted, and Kubernetes operator deployments alike.
The premise follows directly from everything above: purpose-built watchers for the silent failure modes (merge backlog, replication delay, stuck mutations, Keeper session flapping, insert backpressure), and root-cause analysis attached to every alert - what happened, what it affects, and the fix down to copy-paste SQL. A collector reads system tables only, never your data, and nothing changes on your cluster without a human applying it. NeverBlink customers report 25-30% lower database costs and 80-90% less time spent on database management; LinearB, running it against the ClickHouse cluster behind their customer-facing dashboards, saw dashboard queries drop from 8 seconds to subsecond during the engagement.
Managed ClickHouse doesn't remove the need for this layer, either. ClickHouse Cloud solves provisioning, upgrades, backups, and Keeper - but a query that scans a billion rows because the ORDER BY key doesn't match the access pattern is just as slow and just as expensive on Cloud as anywhere else, and managed pricing often makes workload problems cost more, since the easy fix is always scaling up.
Key Takeaways
- ClickHouse defers work by design; its dangerous failures are silent backlogs, not errors. Watch the background machinery, not just query latency.
- Six signals cover most silent failures: parts per partition, source-vs-MV row drift, TTL/disk trends, replication delay, detached parts, and per-shard skew.
- Inserts get delayed at 1,000 parts per partition and rejected at 3,000 - alert well before the first threshold.
- Stale replicas serve reads for up to 300 seconds of lag by default. If read consistency matters, monitor
absolute_delayand tunemax_replica_delay_for_distributed_queries. - Monitoring, observability, and maintenance are separate layers. Pick a monitoring stack for the first, use the system-table queries here for the second, and for the third - continuous expert-level review of what the signals mean - that's what NeverBlink for ClickHouse now does.
