A practitioner's guide to choosing real-time data analysis tools. Covers ClickHouse, Elasticsearch, OpenSearch, Druid, Pinot, and the pipeline and visualization tools that connect them - with honest trade-offs from engineers who operate these systems in production.

Engineering teams are drowning in data. Logs, traces, metrics, clickstreams, transactions - volumes keep growing, and the pressure to analyze everything in real time grows with them. The monolithic BI platform that handled it all? Gone. Modern data analysis stacks are composable: specialized tools optimized for specific workloads, wired together through pipelines, shared by dashboards.

Good tools exist. Plenty of them. The hard part is choosing the right combination. Real-time analytics, log analysis, full-text search, and time-series monitoring have fundamentally different access patterns. A tool that excels at one will disappoint at another. This guide covers the tools we deploy and operate in production - ClickHouse, Elasticsearch, OpenSearch, and the ecosystem around them - with honest assessments of where each works and where it doesn't. For background on how data reaches these analysis tools, see our overview of data pipelines.

Real-Time Analytics: ClickHouse, Druid, and Pinot

Sub-second aggregations across billions of rows - product analytics, ad-tech reporting, financial dashboards - require a purpose-built OLAP engine. Three open-source projects dominate: ClickHouse, Apache Druid, and Apache Pinot. Same goal (fast analytical queries on large datasets), but sharp differences in architecture, operational complexity, and sweet spots.

ClickHouse

ClickHouse stores data in columnar format and uses vectorized query execution, processing multiple values per CPU instruction via SIMD. A single node can scan billions of rows per second on typical analytical queries. Compression ratios of 10-20x are common - columnar storage groups similar data types together, letting type-specific codecs (ZSTD, LZ4, Delta, DoubleDelta) compress aggressively. On the ClickBench benchmark, ClickHouse consistently runs queries 10-100x faster than traditional databases.

The billion-row benchmark against Elasticsearch tells the story clearly: at 10 billion rows on a single AWS c6a.8xlarge (32 cores, 64 GB RAM), ClickHouse used 19x less storage and ran aggregation queries 5-7x faster. At 100 billion rows, ClickHouse stored the dataset in 412 GB. Elasticsearch couldn't load it at all. The hardware cost gap is just as stark - an 8-core ClickHouse instance matched the throughput of a 32-core Elasticsearch setup. That's 4x cheaper hardware for comparable performance.

The SQL dialect is broad and familiar: GROUP BY, window functions, CTEs, approximate aggregations like uniqHLL12, array/map operations for semi-structured data. Uber, Cloudflare, Bloomberg, Deutsche Bank, Spotify, and Sentry all run ClickHouse in production across logging infrastructure, web analytics, experimentation platforms, and financial reporting. ClickHouse Cloud now serves more than 3,000 customers, with ARR growing over 250% year-over-year.

Trade-offs exist. Join performance on very large tables can be unpredictable (broadcast joins by default), and ClickHouse isn't designed for point lookups or transactional workloads. Concurrency targets analytical patterns - tens to low hundreds of concurrent queries, not thousands of lightweight lookups. For aggregation-heavy analytics where raw speed and storage efficiency matter, though, nothing else comes close.

Apache Druid and Apache Pinot

Druid and Pinot were both built for OLAP-style queries with tight real-time ingestion from streaming sources like Kafka. Druid has strong time-series slicing with built-in roll-ups and approximate algorithms (HyperLogLog, data sketches) - Airbnb, Netflix, and Walmart run it in production. Pinot came out of LinkedIn and targets user-facing analytics with extreme query concurrency. At LinkedIn, Pinot handles 200,000+ queries per second at p95 latency under 100ms, powering 50+ user-facing products while ingesting 1.5 million events per second.

Where do they beat ClickHouse? Massive query concurrency. If hundreds of thousands of lightweight queries from end users are hitting dashboards simultaneously, Pinot's architecture was purpose-built for that pattern. Native streaming ingestion from Kafka with sub-second data freshness is another strength - both Druid and Pinot have mature ingestion pipelines here. Pinot also supports real-time upserts, useful for user-facing features where per-row data freshness matters.

Where does ClickHouse win? Operational simplicity. A single binary - no separate broker, controller, coordinator, or minion nodes. Broader SQL support including full JOINs and CTEs. Better single-query performance on complex aggregations. ClickHouse has the fewest moving parts of the three, which directly translates to lower operational overhead. Druid requires multiple node types (coordinator, overlord, broker, historical, middlemanager) and significant developer time to keep running. Pinot uses a microservice architecture with separate controllers, brokers, servers, and minions. Both carry steeper learning curves and higher operational costs than ClickHouse, especially for teams without dedicated platform engineers. Join support is limited in both - Druid supports only lookup joins, and Pinot has restricted join capabilities - compared to ClickHouse's full SQL JOINs.

Log and Event Analysis: Elasticsearch, OpenSearch, and the ClickHouse Alternative

Elasticsearch built its reputation on log analysis and remains the most widely deployed platform for this workload. The landscape has shifted, though. OpenSearch offers a fully open-source alternative, and ClickHouse is gaining traction as a cost-effective backend for high-volume log storage.

Elasticsearch and the ELK Stack

Elasticsearch's inverted index architecture is optimized for full-text search and filtering. The ELK stack (Elasticsearch, Logstash, Kibana) remains the most widely deployed log analysis platform in the industry. The strengths are well-understood: powerful full-text search with relevance scoring, real-time indexing, a rich aggregation framework, and Kibana's visualization capabilities for log exploration.

The weaknesses are just as well-documented. Elasticsearch is storage-hungry compared to columnar databases - inverted indexes, doc values, and stored fields all consume disk. At high log volumes, storage costs grow linearly and can become the dominant infrastructure expense. JVM heap pressure under heavy indexing and query load is a persistent operational headache. And since Elastic changed the license from Apache 2.0 to SSPL in 2021 (later adding AGPLv3 as an option alongside SSPL and ELv2), licensing has become a factor in architectural decisions.

OpenSearch

OpenSearch is the Apache 2.0 fork of Elasticsearch 7.10, maintained by AWS and a growing community of over 400 contributing organizations and 3,300+ contributors. For most log analysis use cases, it offers feature parity with Elasticsearch - indexing, searching, and aggregation all work the same way. The differentiation comes from native plugins: security analytics for SIEM-style workloads with prepackaged detection rules and threat intelligence, an observability plugin with trace and log analytics, and Data Prepper - a server-side data collector with native OpenTelemetry support that handles filtering, enrichment, transformation, and data redaction before anything reaches OpenSearch.

The two projects are diverging. OpenSearch 3.0 (built on Lucene 10) represents a clean break - Elasticsearch plugins don't run on OpenSearch and vice versa. Elastic's commercial ML, APM agents, and Canvas aren't available in OpenSearch. On the flip side, OpenSearch's built-in security plugin (free, based on the former Open Distro), its SQL engine, and its anomaly detection have no Elasticsearch equivalent without a paid license.

OpenSearch makes more sense than Elasticsearch when you need a permissive open-source license (Apache 2.0), when you're running on AWS and want tight managed-service integration, or when you're building security analytics workflows that benefit from built-in SIEM capabilities. For a detailed breakdown, see our OpenSearch vs Elasticsearch comparison.

ClickHouse for Logs

Observability platforms like SigNoz, Uptrace, Qryn, and BetterStack increasingly use ClickHouse as their log storage backend. ClickHouse's own ClickStack initiative positions it as a native observability data platform - OpenTelemetry ingestion with a unified store for logs, metrics, and traces.

The compression numbers are hard to ignore: in published benchmarks, ClickHouse achieved a compression ratio of 19.8x on a log dataset (890 GB uncompressed down to 45 GB compressed), while Elasticsearch needed 51.3 GB for the same data. ClickHouse's own internal "LogHouse" deployment compresses over 10 petabytes of telemetry data down to 600 terabytes - 16x compression. At scale, ClickStack reports 4x lower costs and 10x faster queries compared to Elastic Observability for analytical workloads.

The trade-off: ClickHouse doesn't natively support the fuzzy full-text search that Elasticsearch excels at. It added GA support for inverted text indexes in version 26.2, but these are large (hundreds of megabytes per part) and cover a narrower set of search patterns than Lucene's inverted index. If your log analysis workflow depends on ad-hoc text search with fuzzy matching, Elasticsearch or OpenSearch is the better pick. If your primary access pattern is structured queries, aggregations, and dashboards - and cost matters at high volumes - ClickHouse delivers substantially lower storage and compute costs.

Data Integration and Pipeline Tools

Your analysis tool is only as good as the pipeline feeding it. Getting data reliably from sources into analytical stores is half the battle, and the tooling has matured rapidly.

For batch and ELT workloads, Fivetran and Airbyte dominate. Fivetran offers 740+ fully managed connectors for ELT from SaaS apps, databases, and cloud storage. It supports ClickHouse Cloud as a destination and can extract from both Elasticsearch and OpenSearch as sources. One caveat: Fivetran's pricing shifted to per-connector Monthly Active Rows (MAR) billing in 2025, and some organizations reported 70% cost increases after the change. Airbyte provides 600+ connectors with an open-source core (Elastic License 2.0 for connectors, MIT for the protocol). It supports self-hosted deployment including fully air-gapped environments - critical in regulated industries like finance and healthcare. Airbyte moved to capacity-based "Data Workers" pricing in 2025, decoupling cost from data volume. If you need to build or customize connectors, Airbyte's open CDK and integration with Airflow, Dagster, and dbt give engineers more control.

For real-time streaming, Apache Kafka plus Apache Flink is the standard architecture. Kafka handles event streaming and buffering. Flink performs stateful transformations - filtering, enriching, aggregating, windowing - before data lands in ClickHouse (via ClickPipes or Kafka connectors) or OpenSearch (via Flink's native sink connector). This combination handles millions of events per second with sub-second end-to-end latency. For simpler cases that don't need Flink's stateful processing, ClickHouse can consume directly from Kafka using its built-in Kafka table engine, with materialized views handling light transformations on ingestion.

OpenTelemetry (CNCF graduated project) is now the de facto standard for collecting observability data. The second most active CNCF project after Kubernetes. Vendor-neutral APIs and SDKs for traces, metrics, and logs across 11+ languages. Compatible with over 40 observability backends. Every major vendor supports it - Datadog, Splunk, New Relic, Elastic, Grafana Labs, and dozens more. If you're building an observability pipeline today, OTel is the collection layer. The question is which backend you send data to. For a hands-on walkthrough, see our practical guide to OpenTelemetry with OpenSearch.

Visualization and Dashboarding

Raw query results need a presentation layer. The right visualization tool depends on whether you're building operational dashboards or business analytics.

Grafana is the open-source standard for operational monitoring. Native connections to ClickHouse, Elasticsearch, OpenSearch, Prometheus, and 30+ other data sources via built-in support and plugins. The official ClickHouse plugin for Grafana (v4.x) is actively maintained by the ClickHouse and Grafana teams - first-class support for logs and traces, OpenTelemetry schema auto-detection, and a no-SQL-required query builder for observability data. Where Grafana shines: time-series dashboards, alerting, and correlating data across multiple sources in a single view.

Kibana is tightly coupled with Elasticsearch and strongest for log exploration and search-driven analytics. If Elasticsearch is your primary analysis tool, Kibana's Discover, Lens, and Dashboard features are hard to replace. The tight integration makes search-driven workflows feel native in a way no multi-source tool can match. OpenSearch Dashboards offers equivalent functionality for OpenSearch deployments, including built-in observability and security analytics dashboards. Same constraint for both: single-source tools locked to their respective ecosystems.

Apache Superset fills the business intelligence gap. SQL-native, connects to ClickHouse and most analytical databases via SQLAlchemy (PostgreSQL, BigQuery, Snowflake, Druid, and dozens more), with chart types and dashboards oriented toward business metrics rather than operational monitoring. Commercial alternatives like Tableau, Looker, and Metabase offer more polish and less operational overhead, but at a licensing cost that open-source Superset avoids.

In practice: Grafana for operational monitoring and alerting, Kibana/OpenSearch Dashboards for search-driven log exploration, Superset (or a commercial BI tool) for business analytics. Many organizations run two or all three. They solve different problems and coexist naturally.

How to Choose: A Decision Framework

A practical matrix based on workload type:

Workload Best Tool Why
Sub-second analytics on billions of rows ClickHouse Fastest analytical query engine, best compression
Full-text log search and exploration Elasticsearch / OpenSearch Inverted index optimized for text search
High-volume log storage (cost-sensitive) ClickHouse 12-19x better compression than Elasticsearch
Real-time dashboards (high concurrency) ClickHouse or Druid Both handle concurrent dashboard queries well
Security event analysis (SIEM) OpenSearch Native security analytics plugin, SIEM integrations
Observability (traces + metrics + logs) OpenSearch + ClickHouse OpenSearch for traces/logs, ClickHouse for metrics

Beyond the workload itself, operational factors matter:

Team expertise. Elasticsearch and OpenSearch have larger talent pools and more community resources. ClickHouse expertise is growing fast - 900+ verified production deployments as of 2025 - but still rarer. Druid and Pinot require specialized knowledge.

Managed vs. self-hosted. AWS OpenSearch Service, Elastic Cloud, and ClickHouse Cloud all reduce operational burden. Elastic Cloud starts at ~$99/month for small deployments. ClickHouse Cloud charges ~$0.22-0.39 per compute-unit-hour and $25.30/TB-month for storage. AWS OpenSearch Service bills per instance-hour with reserved instance discounts up to 52%.

Cost trajectory. This is where choices diverge most. ClickHouse is the most cost-effective at scale - superior compression, lower compute requirements. Elasticsearch and OpenSearch costs grow linearly with data volume. Commercial SaaS observability tools sit at the other extreme: Datadog charges $0.10/GB for log ingestion plus $1.70/million events for indexing, and Splunk's enterprise licensing runs $1,800-2,500/GB/day annually. At 1.5 TiB of logs, organizations routinely see 23x cost multiples for Elastic Cloud and 42x for Datadog compared to a ClickHouse-based stack.

Migration path. Many organizations start with Elasticsearch or OpenSearch for logs and add ClickHouse for analytics as data volumes grow and cost pressure mounts. A well-trodden path - the two systems complement each other rather than compete directly.

Key Takeaways

No single "best" data analysis tool exists. The most effective architectures combine specialized systems: ClickHouse for analytics and cost-effective log storage, OpenSearch or Elasticsearch for full-text search and log exploration, OpenTelemetry for collection, Grafana or Kibana for visualization. The right combination depends on your workload mix, data volume, team expertise, and budget.

Getting that architecture right - and operating it at production scale without burning through your infrastructure budget - that's the real engineering challenge. At BigData Boutique, we specialize in ClickHouse, Elasticsearch, and OpenSearch. We help teams evaluate, deploy, and optimize their data analysis infrastructure based on what actually works in production, not vendor marketing.