A practitioner comparison of anomaly detection algorithms for observability data: statistical baselines, Isolation Forest, Random Cut Forest, and Prophet, and which one fits the shape of your logs and metrics.

Anomaly Detection Algorithms for Logs and Metrics: Isolation Forest, Random Cut Forest, Prophet, and When Each Works

Most teams reach for machine learning the moment someone says "anomaly detection." That instinct is often wrong. For a single metric with a stable daily rhythm, a z-score against a rolling mean catches the same spikes that an ensemble of trees would, runs in microseconds, and never surprises you at 3 a.m. with a model that drifted. The hard part is matching the algorithm to the shape of your data and to the cost you are willing to pay for a false positive.

This post compares four families of anomaly detection algorithms that show up repeatedly in observability work: statistical baselines (moving average, z-score, EWMA), Isolation Forest, Random Cut Forest (RCF), and Prophet. Each one assumes something different about your data: whether it streams or arrives in batches, whether it has seasonality, whether it is one dimension or many. Pick the wrong assumption and you get alert fatigue or silent misses.

The three categories you are actually choosing between

Anomaly detection algorithms split into three groups based on how they model "normal." Statistical methods model normal as a distribution and flag points that fall too far from it. Tree-based outlier detectors model normal as a region of feature space and flag points that are easy to isolate from the rest. Forecasting methods model normal as a prediction and flag points where reality diverges from the forecast.

That framing tells you what each method can and cannot see. A z-score on a single time series has no concept of "Tuesday looks different from Sunday." Prophet does. Isolation Forest has no concept of order in time at all; shuffle the rows and the scores are statistically unchanged. Random Cut Forest is built specifically to handle order, because it maintains a sketch of a stream and updates it point by point.

The other axis is dimensionality. A CPU utilization metric is univariate. A request that carries latency, status code, payload size, and upstream host is multivariate, and an anomaly may live in the combination of values rather than in any single field. The methods below differ sharply in how they handle that.

Statistical baselines: where most metric monitoring should start

For a univariate metric, the workhorses are moving average, z-score, MAD (median absolute deviation), and EWMA (exponentially weighted moving average). A z-score measures how many standard deviations a point sits from a rolling mean. MAD swaps mean and standard deviation for median and median deviation, which makes it far more robust when your history already contains spikes. EWMA weights recent observations more heavily than old ones, so the baseline adapts as traffic grows.

For a large class of metrics these are the correct choice, not a compromise. They are interpretable: an on-call engineer can read "value was 6 sigma above the 1-hour mean" and act on it. They cost almost nothing to compute, which matters when you are scoring thousands of series every few seconds. And they have no training phase to drift or go stale.

In ClickHouse a rolling z-score is a window function over the metric table:

SELECT
      ts,
      value,
      (value - avg(value) OVER w) / nullIf(stddevPop(value) OVER w, 0) AS zscore
  FROM metrics
  WHERE name = 'http_requests_per_sec'
  WINDOW w AS (ORDER BY ts ROWS BETWEEN 59 PRECEDING AND 1 PRECEDING)
  QUALIFY abs(zscore) > 4
  ORDER BY ts DESC

The window excludes the current row so a spike does not inflate its own baseline. QUALIFY needs ClickHouse 24.4 or later; on older versions wrap the query in a subquery and filter with WHERE. In Prometheus the same idea is a recording rule plus an alert:

groups:
    - name: zscore
      rules:
        - record: instance:requests:zscore_1h
          expr: |
            (rate(http_requests_total[5m]) - avg_over_time(rate(http_requests_total[5m])[1h:1m] offset 10m))
            / stddev_over_time(rate(http_requests_total[5m])[1h:1m] offset 10m)
        - alert: RequestRateAnomaly
          expr: abs(instance:requests:zscore_1h) > 4
          for: 5m

The baseline is offset by 10 minutes so a sustained spike does not enter its own baseline before for elapses.

A plain z-score assumes the series is roughly stationary, so a metric with strong daily or weekly seasonality will throw false positives every morning when traffic ramps and miss real problems during quiet hours. The usual fix is to deseasonalize first (compare each point to the same hour last week, or subtract a seasonal baseline) and then apply the statistic. In PromQL that is offset 1w. If you find yourself building an increasingly elaborate seasonal correction on top of a z-score, that is the signal to move to a forecasting model.

A standard playbook for metric alerting:

  1. Start with EWMA or a rolling z-score on the raw series.
  2. If mornings and weekends cause false alarms, deseasonalize against the same period last week before scoring.
  3. Switch from mean/std to median/MAD if your history is spiky.
  4. Only reach for ML once you have a concrete shape that statistics cannot capture: strong multi-period seasonality, multivariate interactions, or high cardinality.

Isolation Forest and Random Cut Forest: tree-based outlier detection

Isolation Forest, introduced by Liu, Ting, and Zhou in 2008, builds an ensemble of random trees. At each node it picks a feature at random and a split value at random. Anomalies, being few and different, get isolated into their own leaf after only a handful of splits, so a short average path length across the forest signals an outlier. The scikit-learn IsolationForest implementation makes this a few lines of Python. It is unsupervised, fast to train, and indifferent to the scale of individual features because every split is per-feature and random. It does degrade when many of the features are irrelevant noise, a limitation the original paper notes, so prune the feature set before trusting the scores.

The catch is that Isolation Forest is a batch method with no notion of time. It treats each row as an independent point in feature space. That is ideal for finding outlier log events or unusual request fingerprints across a fixed window, and it handles the multivariate case naturally. It is the wrong tool for a streaming metric where "anomalous" means "different from five minutes ago," because shuffling the timestamps would not meaningfully change any score.

Random Cut Forest is the streaming answer to that gap. The Robust Random Cut Forest paper by Guha, Mishra, Roy, and Schrijvers (ICML 2016) adapts the isolation idea to data streams by maintaining a forest as a continuously updated sketch of the input. As each new point arrives, RCF estimates how much that point disturbs the existing tree structure and emits an anomaly score. It was designed for streaming, high-dimensional data and degrades gracefully on duplicates and near-duplicates, which are common in metric streams.

Random Cut Forest in OpenSearch Anomaly Detection

RCF is the algorithm behind OpenSearch Anomaly Detection and AWS managed anomaly detection features. In OpenSearch it runs unsupervised over your indexed time series, computing an anomaly grade and a confidence score per interval. RCF does not model seasonality explicitly. What it sees is a shingle: the last N consecutive values packed into one vector, so daily shape enters the model only through the window of recent points. The shingle_size knob (default 8, range 1 to 128) is the lever here: smaller shingles react faster and produce more false positives, larger ones smooth noise and take longer to flag a real change. The plugin adapts to drift and organic growth in a way static thresholds do not, but it does not learn "Mondays are different" the way Prophet does.

The high-cardinality mode is the part most teams underuse. Set a category field (host, IP, customer, product ID) and OpenSearch builds a separate per-entity model, so each entity gets its own baseline rather than being smeared into a global average. A per-host latency detector looks like this:

PUT _plugins/_anomaly_detection/detectors
  {
    "name": "p99-latency-per-host",
    "indices": ["metrics-*"],
    "time_field": "@timestamp",
    "detection_interval": { "period": { "interval": 5, "unit": "Minutes" } },
    "window_delay":       { "period": { "interval": 1, "unit": "Minutes" } },
    "shingle_size": 8,
    "category_field": ["host.name"],
    "feature_attributes": [{
      "feature_name": "p99_latency",
      "feature_enabled": true,
      "aggregation_query": {
        "p99": { "percentiles": { "field": "latency_ms", "percents": [99] } }
      }
    }],
    "result_index": "opensearch-ad-plugin-result-latency"
  }

Entity models share an in-heap cache capped by plugins.anomaly_detection.model_max_size_percent (10% of heap by default); models that do not fit are loaded from disk on demand, with hot and recent entities prioritized. AWS measured this hybrid scheme at one million entities in ten minutes back in 2022, on RCF 2.0, using synthetic data on an otherwise idle cluster, so expect the ceiling to be memory and disk I/O rather than the algorithm. For per-host or per-tenant observability, that per-entity isolation is usually what separates a useful detector from a noisy one.

False-positive behavior has also moved. OpenSearch 2.17 shipped four RCF changes (statistical tracking of candidate anomalies, grouped alerting, approximate nearest-neighbor rescoring, and expected-value rescoring) that the AD team measured on five Numenta Anomaly Benchmark AWS CloudWatch series: false positives fell from 106 in OpenSearch 2.9 to 6 in 2.17 (a 94.3% reduction), with precision rising from 5.4% to 53.8%. That is a small sample, but numbers from a cluster older than 2.17 do not carry over. Recent versions also apply a default suppression rule that ignores deviations under 20% from the expected value; override it with the rules field if you need tighter sensitivity.

Prophet and forecasting-based detection: when seasonality is the whole story

Prophet is an open-source forecasting library from Meta. It fits an additive model that decomposes a series into trend, seasonality (yearly, weekly, daily, plus custom periods you define), and holiday effects. To use it for anomaly detection you forecast the expected value with an uncertainty interval, then flag any actual observation that lands outside that interval. The anomaly is defined relative to what the model expected, not relative to a flat rolling mean.

This is the right family when seasonality and known calendar events dominate your signal. Business-hours traffic, weekend dips, end-of-month batch jobs, Black Friday: Prophet models these explicitly, including the ability to register a holiday calendar so a predictable spike does not page anyone. It is robust to missing data and to occasional outliers in the training history, and it does not demand a time-series specialist to operate. Classical alternatives in this family (ARIMA, SARIMA, Holt-Winters) handle similar seasonal structure and are lighter weight, but they require more care in selecting orders and differencing. One caution: Prophet is actively maintained again (1.5.0 shipped in October 2026), but pin versions; its Stan backend has caused install pain in the past.

Prophet is a batch fit; you retrain periodically rather than scoring a live stream point by point. It wants several seasonal cycles of history to learn the pattern, so it is poor at cold start and on young metrics. It forecasts a single target series; you can add external regressors, but it will not detect anomalies that live in the combination of several signals. And on a metric with no real seasonality it adds complexity and tunable knobs without buying you anything a z-score would not. Reserve it for series where the seasonal shape is the dominant feature and where calendar-aware suppression of known spikes is worth the operational overhead of retraining.

Logs are a different problem

Everything above assumes numeric series. Raw log lines are not, and running any of these algorithms on raw text produces noise. The working pipeline is two steps. First, collapse lines into templates: Drain-style parsers replace variable tokens (IDs, timestamps, hostnames) with placeholders so Connection to 10.0.3.7 timed out after 30s and Connection to 10.0.9.2 timed out after 31s become one template. Second, run detection on the structured result. Two shapes work well:

  • Template frequency as a metric. Count each template per minute and feed the counts to the same z-score or RCF machinery, per-template or per-service via a category field. A new template appearing, or a known one spiking, is the usual outage signature.
  • Outlier events in a batch. Encode each event's template ID, severity, source, and a few numeric fields, and run Isolation Forest over a window. This finds the individually weird line rather than the volume change.

Elastic's ML jobs and OpenSearch's log-pattern tooling both automate parts of this; the algorithmic choice underneath is the same.

The comparison table and how to choose

None of these wins outright. They occupy different corners of a design space defined by streaming versus batch, one dimension versus many, and whether seasonality is present.

Algorithm Supervised? Streaming or batch Seasonality handling Dimensionality Typical use
Z-score / EWMA / MAD Unsupervised Streaming None (deseasonalize manually) Univariate Default for stable single metrics; cheap, interpretable alerting
Prophet (forecasting) Unsupervised Batch (periodic retrain) Strong, explicit (daily/weekly/yearly + holidays) Univariate Seasonal business metrics with calendar effects
Isolation Forest Unsupervised Batch None (no time concept) Multivariate Outlier rows: unusual log events, request fingerprints over a window
Random Cut Forest Unsupervised Streaming Adapts to drift and trend; seasonality only implicitly via shingling Multivariate, high-cardinality Live metric/log streams; per-entity detection in OpenSearch

Rules of thumb:

  • Single metric, stable or mildly trending: start with a rolling z-score or EWMA. Do not skip to ML.
  • Single metric, strong daily/weekly cycle and known calendar events: Prophet, or a deseasonalized statistic if you want to stay lightweight.
  • A live stream where you cannot retrain offline, especially many entities: Random Cut Forest, with a category field for host- or tenant-level models.
  • Outlier records in a batch, order does not matter: Isolation Forest.

Readers will ask about three other options. Elastic's machine learning jobs are the Elasticsearch counterpart to OpenSearch AD: unsupervised time-series models with per-entity splitting, a similar fit. Prometheus predict_linear is a trend extrapolation, useful for "disk fills in 4 hours" alerts, not an anomaly detector. And LLMs do not belong in the scoring loop for either metrics or logs; they are useful for summarizing an incident after a detector fires, not for deciding whether one should.

Whatever you pick, the production failure mode is almost always alert fatigue. A detector that fires on every Monday morning ramp gets muted within a week, after which it might as well not exist. Tune thresholds against a labeled window of past incidents, suppress known seasonal and deployment-related spikes, and require a minimum confidence or a sustained breach before paging. If you are still deciding where logs and metrics should live before any of this, our guides to choosing log management tooling and monitoring Amazon OpenSearch cover the platform layer, and the OpenTelemetry with OpenSearch guide covers getting the signals in cleanly in the first place.

If you are weighing these approaches against the noise problem in your own stack, BigData Boutique builds and tunes anomaly detection on OpenSearch and ClickHouse for production observability workloads.