Itamar Syn-Hershko
ClickHouse observability means watching the background machinery - merges, TTL, replication, Keeper - not just query latency. The silent failure modes of ClickHouse, the system-table queries that surface them early, and where AI-assisted maintenance fits.
The worst ClickHouse incidents don't start with an alert. A merge backlog builds for days, a materialized view quietly drops rows, a replica serves stale reads - and every dashboard stays green until ingestion halts or someone notices the numbers don't add up.
ClickHouse observability is the practice of tracking the database's deferred background work - merges, mutations, TTL cleanup, replication, and Keeper coordination - rather than only the query path. The distinction matters because ClickHouse is fast precisely because it defers work, and deferred work fails silently. This post walks through the failure modes that hide from standard dashboards, the system tables that expose them, and what to do once you can see them. If you're choosing a monitoring stack first, start with our comparison of ClickHouse monitoring tools and come back here for what to actually watch.
Why ClickHouse Fails Quietly
Most databases do their expensive work at write time. ClickHouse inverts this: an INSERT just writes an immutable part to disk, and everything else - merging parts, applying TTL deletes, replicating data, materializing mutations - happens later, in background pools, eventually. That design is why a modest cluster ingests millions of rows per second.
It's also why the failure surface looks nothing like a typical OLTP database. When background machinery stalls, foreground operations keep succeeding. Inserts still return OK while parts pile up. Queries still answer while a replica drifts behind. Disk usage creeps while TTL cleanup silently stops. There is no error to page on, because from ClickHouse's perspective nothing has failed yet - work is just pending. The gap between "pending" and "outage" is where observability earns its keep.
ClickHouse is honest about all of this: it exposes over a thousand metrics through system tables like system.parts, system.merges, system.replicas, and system.mutations. The problem is rarely missing data. It's knowing which handful of signals predict outages, and checking them continuously.
Five Silent Failure Modes and Their Early Signals
Each of these comes from real production incidents we've worked on across ClickHouse consulting engagements. None of them shows up on a CPU/memory dashboard until late in the game.
| Failure mode | Why it's silent | Early signal |
|---|---|---|
| Merge backlog | Inserts are throttled first, and still succeed | MaxPartCountForPartition trending upward while the background merge pool runs at full capacity |
| Materialized view chains fail | MV writes are not atomic with the source insert; retries and cascades multiply the damage | Failed inserts on tables with MVs, and row-count drift between source and targets |
| TTL stops keeping up | TTL runs only inside merges, at low priority | Table size growing while insert bytes stay flat |
| Replica divergence | Stale replicas keep serving reads by default | absolute_delay in system.replicas |
| Shard skew | Cluster-wide averages mask the hot shard | Per-shard part counts and insert rates diverging |
Merge backlog. Parts accumulate faster than the merge pool clears them. ClickHouse delays inserts at 1,000 parts per partition and rejects them at 3,000 with TOO_MANY_PARTS - and because the delay phase is invisible to callers, the first symptom anyone notices is often the outage itself. The early signal is the combination: MaxPartCountForPartition climbing while BackgroundMergesAndMutationsPoolTask sits at its limit, meaning the pool is saturated rather than idle. And don't assume merges are merely slow - they can be failing outright. A classic case is an AggregatingMergeTree with a uniqExact state over a huge-cardinality column, where a single merge can't fit in memory and is retried forever; check system.merges for entries whose elapsed keeps resetting and system.part_log for MergeParts events with a non-zero error. Watch parts per partition directly:
SELECT database, table, partition, count() AS parts
FROM system.parts
WHERE active
GROUP BY database, table, partition
ORDER BY parts DESC
LIMIT 10;
We cover causes and fixes in depth in the too many parts problem.
Materialized view chains failing. An insert into a source table and the writes into its materialized views are not atomic. By default a failure in any MV in the chain fails the whole insert (unless materialized_views_ignore_errors is explicitly enabled), so the client sees an error - but by then some of the MVs, and possibly the source table, have already committed their rows. The client retries, those targets get the rows a second time, and with several cascading views hanging off one source the inconsistency compounds with every retry. This is a well-documented failure scenario, and it's silent because the retry eventually succeeds and every dashboard shows green. Watch for insert failures on tables that have MVs attached, and run periodic row-count reconciliation between source and MV targets; see our materialized views guide for how cascades behave under failure.
TTL falling behind. TTL deletes only happen inside merges, and TTL-driven merges are rate-limited (merge_with_ttl_timeout defaults to four hours per table) and deprioritized when the pool is busy. Retention quietly stops working, and disk usage climbs until a replica goes read-only. The conscious signal is a divergence: insert bytes per day are flat, yet the table's bytes_on_disk in system.parts keeps growing. With steady ingestion and working retention, table size should plateau - if it doesn't, TTL isn't keeping up. Track that trend per table alongside free space in system.disks and the merge queue.
Replica divergence. A Keeper hiccup leaves a replica stale, and by default distributed queries keep reading from replicas that lag by up to 300 seconds (max_replica_delay_for_distributed_queries). The same query returns different numbers depending on which replica answers. Meanwhile a replica that loses its Keeper session altogether goes read-only:
SELECT database, table, is_readonly, absolute_delay, queue_size
FROM system.replicas
WHERE is_readonly OR absolute_delay > 30;
More on this in ClickHouse replication health.
Shard skew. A skewed sharding key piles writes onto one shard. Cluster averages look fine while that shard hits TOO_MANY_PARTS first - and ClickHouse has no automatic rebalancing to save you. Always chart per-shard, never just per-cluster.
Monitoring, Observability, and Maintenance Are Different Jobs
These terms get used interchangeably, and the sloppiness has a real cost: teams buy a monitoring stack and believe they've bought the other two.
| Layer | Question it answers | Typical tooling |
|---|---|---|
| Monitoring | Is a metric outside its threshold right now? | Grafana, Datadog, the built-in /dashboard |
| Observability | Can I explain why from the signals I collect? | System tables, query_log analysis, per-shard views |
| Maintenance | Is someone acting on it before it becomes an incident? | Runbooks, expert review, AI-assisted platforms |
Monitoring tells you a partition crossed 800 parts. Observability connects that to the single-row inserts from a new service that bypassed your batching layer. Maintenance is enabling async_insert for that client, or fixing the batch size at the source - before the 1,000-part throttle kicks in. Our ClickHouse monitoring tools comparison covers the first layer well; the queries above get you much of the second. The third has traditionally meant having a ClickHouse expert on staff, which is exactly the scarce resource most teams are missing.
Closing the Loop with AI-Assisted Maintenance
The maintenance layer is the one we kept getting pulled into as consultants, and it's why our team built NeverBlink - which now supports ClickHouse alongside Elasticsearch and OpenSearch, covering ClickHouse Cloud, self-hosted, and Kubernetes operator deployments alike.
The premise follows directly from everything above: purpose-built watchers for the silent failure modes (merge backlog, replication delay, stuck mutations, Keeper session flapping, insert backpressure), and root-cause analysis attached to every alert - what happened, what it affects, and the fix down to copy-paste SQL. A collector reads system tables only, never your data, and nothing changes on your cluster without a human applying it. NeverBlink customers report 25-30% lower database costs and 80-90% less time spent on database management; LinearB, running it against the ClickHouse cluster behind their customer-facing dashboards, saw dashboard queries drop from 8 seconds to subsecond during the engagement.
Managed ClickHouse doesn't remove the need for this layer, either. ClickHouse Cloud solves provisioning, upgrades, backups, and Keeper - but a query that scans a billion rows because the ORDER BY key doesn't match the access pattern is just as slow and just as expensive on Cloud as anywhere else, and managed pricing often makes workload problems cost more, since the easy fix is always scaling up.
Key Takeaways
- ClickHouse defers work by design; its dangerous failures are silent backlogs, not errors. Watch the background machinery, not just query latency.
- Five signals cover most silent failures: parts per partition against merge-pool saturation, failed inserts and row drift on tables with materialized views, table size growing while insert bytes stay flat, replication delay, and per-shard skew.
- Inserts get delayed at 1,000 parts per partition and rejected at 3,000 - alert well before the first threshold.
- Stale replicas serve reads for up to 300 seconds of lag by default. If read consistency matters, monitor
absolute_delayand tunemax_replica_delay_for_distributed_queries. - Monitoring, observability, and maintenance are separate layers. Pick a monitoring stack for the first, use the system-table queries here for the second, and for the third - continuous expert-level review of what the signals mean - that's what NeverBlink for ClickHouse now does.
