Itamar Syn-Hershko
A practitioner's guide to LLM evaluation: the layered system of offline regression, online scoring and human calibration, the metrics that map to business risk, and a 2026 comparison of DeepEval, Ragas, Promptfoo, LangSmith, Braintrust, Phoenix, Langfuse, Opik, and MLflow.
Most teams treat LLM evaluation as a tool-selection problem. Picking DeepEval over Ragas, or LangSmith over Braintrust, is downstream of a question almost nobody answers explicitly: what does the evaluation system look like? Shipping reliable LLM features requires a layered system, and frameworks slot into that architecture. They do not replace it.
This post lays out the architecture first, then the CI wiring that makes it bite, and only then maps the 2026 framework landscape onto it. It assumes you have already shipped or prototyped an LLM-powered feature and want to stop being surprised by it.
Why LLM Evaluation Is Not Like Traditional ML Evaluation
Classic supervised ML had the convenient property that a single number (accuracy, F1, AUC) summarized model quality on a frozen test set. LLM applications break that on every axis. Outputs are open-ended, sampling is stochastic, and the same prompt can produce different answers across runs. The "test set" is moving because prompts and upstream models change, not just data.
Cost and latency belong on the same scorecard as accuracy. A model that is two points more accurate but three times slower or five times more expensive is rarely the right answer for a user-facing surface.
Public benchmarks like MMLU-Pro, GPQA, BIG-Bench Hard, and SWE-bench Verified tell you which model is generally smarter. They do not tell you whether your support copilot stops fabricating policy numbers or your coding agent stops over-using a deprecated API. Those are application-level questions, and a benchmark of grad-school physics multiple choice will not answer them.
What to Actually Evaluate: Metrics That Map to Business Risk
The first decision is what failure modes hurt you, not what is easy to measure.
Task metrics are the closest analog to classical ML and the easiest to automate: exact match, structured-output validity (does the JSON parse, does it conform to the schema), tool-call correctness for agents. When retrieval is in the loop, Ragas has standardized a useful taxonomy: faithfulness (is the answer grounded in retrieved context), answer relevancy (does the answer address the query), context precision (is the retrieved context focused), context recall (does the retrieved context contain the answer). All four are computable without ground-truth labels using an LLM judge.
Agents add trajectory correctness, tool-selection accuracy, and step efficiency. Most frameworks now ship these as first-class scorers, including DeepEval, Langfuse, Braintrust, MLflow (ToolCallCorrectness) and the Azure AI evaluators; if your agents do non-trivial multi-step reasoning, you need them. Safety and policy checks (toxicity, PII leakage, jailbreak resistance, prompt-injection robustness) are usually classifier-based and run on every output regardless of task.
Operational metrics are p50 and p95 latency, tokens per request, and cost per successful task. The denominator matters: failed attempts inflate cost-per-completion, which is what shows up on the bill. Human-calibrated metrics (preference, helpfulness, tone) are expensive to collect, but they anchor the automated layers. Without them, you are optimizing a proxy you have not validated.
The Layered Evaluation System
Evaluation is three loops, not one.
Layer 1, offline regression. A versioned golden dataset (typically 100-500 examples), deterministic or LLM-judge scorers, run in CI on every PR. Catches regressions before they ship. This is where DeepEval, Promptfoo, Ragas, and similar libraries live.
Layer 2, online and shadow evaluation. Sampling production traffic for live scoring, A/B tests, interleaved tests, shadow deploys for new prompt or model versions. This is where hosted platforms (LangSmith, Phoenix, Langfuse, Braintrust, Opik) earn their keep, because they couple tracing infrastructure with eval harnesses.
Layer 3, human calibration. A small expert-labeled set that anchors what "good" means. The other two layers drift without it; LLM judges are biased toward verbosity and self-similarity. In practice this is 100-300 examples, labeled by two domain raters plus an adjudicator, refreshed quarterly and any time the product changes what "good" means. Each refresh, run the LLM judge over the same set and look at where it disagrees with the humans; every disagreement cluster is either a rubric fix or a scorer change. If the judge agrees with humans on 95% of the set but the 5% is concentrated in one intent, that intent needs its own scorer.
The layers feed each other. Production failures surfaced by Layer 2 become regression cases in Layer 1. Layer 3 disagreements with the LLM judge force scorer revisions. A team running only Layer 1 ships well-tested regressions of a quality target nobody validated. A team running only Layer 2 catches problems after users see them.
Wiring Evals Into CI/CD and Production
The eval system is only useful if it blocks bad changes from shipping.
Pre-merge, a fast subset of the golden set runs on every PR with threshold scorers as gates: "faithfulness must be at least 0.85", "p95 latency must be below 2 seconds", "cost per request must not increase by more than 10%". Slow scorers (LLM judges) run nightly or pre-release rather than per-PR. Pre-release, the full golden set runs along with the cost and latency budgets and any safety or red-team suites, and a failure closes the release. Post-release, sample 1-5% of production traffic into the eval pipeline, watch judge-score distributions for drift, alert when the tail of low-confidence outputs grows, and auto-promote new failure clusters into the regression set so the next release is tested against today's bugs.
Release strategies that pay off: shadow deploys (new version sees production traffic, output discarded, scored offline), canary releases on traffic slices, and feature-flagged prompt and model changes so rollback is one toggle rather than a redeploy.
The Golden Dataset Decides Everything
Evaluation frameworks are commodity. Golden datasets are not. The highest-leverage activity in an evaluation program is curating a few hundred high-quality examples that represent your real distribution.
Real production traces beat synthetic every time. If you do not have production traffic yet, prompt-generated examples are a starting point but should be replaced as soon as real data exists. Stratify by intent, difficulty, and observed failure mode; random sampling under-represents the long-tail failures that matter.
Label with two raters plus adjudication and explicit rubrics. Track inter-annotator agreement (Cohen's kappa or simple percent agreement). As a rule of thumb, if your raters land below about 0.7 kappa, the rubric is the problem, not the model.
Treat eval data as code: version it in the same repo as the application, gate changes through PRs, and snapshot it whenever you change scoring logic so old runs remain comparable. Datasets that drift silently invalidate every comparison you make.
LLM-as-a-Judge: Where It Works and Where It Fails
Judge models are seductive because they collapse a hard labeling problem into an API call. They also have well-documented failure modes: position bias (preferring whichever response is shown first), verbosity bias (preferring longer answers), self-preference (a model rates its own outputs higher), and reference leakage when the judge has seen the test data during pretraining.
The original G-Eval paper showed that chain-of-thought-style scoring with explicit rubrics improves correlation with human judgment, but does not eliminate the biases. The LLM-as-a-Judge paper (the MT-Bench work) measured position bias empirically. Since then, the practical recipe has stabilized: use a stronger judge than your generator, anchor judgments with rubrics rather than free-form scoring, randomize position in pairwise comparisons, and audit a sample against humans regularly.
Judges work well on structured criteria (does the output cite at least one source, does the JSON parse, does the response stay on-topic), pairwise preferences with anchored rubrics, and faithfulness checks against retrieved context where the judge has the source material in the prompt. They fail on subjective tone, anything where verbosity correlates with perceived quality, multi-turn coherence, and high-stakes safety judgments. We have written about a specific failure mode in Thinking Fast and Failing Slow: judges look fine on aggregate metrics while missing the cases that matter.
The mitigations: break the judgment into 3-5 binary checks rather than a 1-10 score, use pairwise comparisons rather than absolute scores, run the judge twice with positions swapped and discard ties, and treat any judge score as a hypothesis until it correlates with a human-labeled holdout.
Framework Landscape, Mapped to the Layers
What follows is the 2026 picture, organized by which layer each tool targets. For a feature-by-feature comparison of the hosted platforms, see our LLM observability tools post; this section only covers how they map onto the layers.
Layer 1 tools. DeepEval is the closest thing to "pytest for LLMs": tests as Python functions, decorators that wrap LLM calls, more than fifty built-in metrics including G-Eval, the Ragas suite and agentic scorers. Ragas is RAG-native and remains the cleanest taxonomy for retrieval evaluation; many teams import its metrics into DeepEval or directly into a hosted platform. Promptfoo is declarative YAML-driven prompt regression with first-class red-team support; OpenAI acquired it in March 2026 and it remains MIT-licensed. The open-source openai/evals repo sees only sporadic commits and its README now points to the hosted Evals product in the OpenAI dashboard; use it as a reference for the YAML registry pattern, not as a starting point.
Layer 2 platforms. LangSmith, Braintrust, Arize Phoenix, Langfuse, and Opik all offer the same shape: tracing capture, dataset management, online evaluators, dashboards, and human review queues. They differ in licensing, self-host options, framework integrations, and pricing. Langfuse is MIT (except its ee/ folders) and self-hosts cleanly. Phoenix self-hosts too, but under the Elastic License 2.0, which is source-available rather than OSI open source and forbids offering it as a managed service; check that against your procurement rules before assuming "open source". Braintrust and LangSmith are commercial-first with stronger out-of-the-box scoring and dataset tooling, and both can run in your own cloud: Braintrust as a self-hosted data plane with Terraform for AWS, GCP and Azure, LangSmith as an Enterprise-plan add-on on your Kubernetes. Opik (Comet) sits in between, with an open-source core and hosted SaaS.
Enterprise and cloud-native. MLflow GenAI evaluation (mlflow.genai.evaluate) is the right answer for teams already on Databricks or running MLflow as their experiment-tracking spine. It ships a set of built-in judges covering correctness, retrieval groundedness and relevance, tool-call correctness, safety and guideline adherence, with results landing in the same MLflow runs as classical model artifacts. Vertex AI Gen AI Evaluation and the Microsoft Foundry SDK (azure-ai-projects 2.0, GA March 2026, folds agents, inference and evaluations into one client; the older azure-ai-evaluation package is the Foundry-classic path) earn their place when compliance, data residency, or the rest of the ML stack is already on that cloud.
Research and model selection. Stanford HELM, EleutherAI's lm-evaluation-harness, MMLU-Pro, GPQA, BBH, and SWE-bench Verified are the right tools for picking a base model and the wrong tools for evaluating whether your application works. Conflating the two is the most expensive mistake we see in evaluation strategy: a team picks the top model on a leaderboard, skips application-level evals, and discovers the regression in their own domain from support tickets.
| Framework | Primary layer | RAG support | Self-host | CI fit | Best for |
|---|---|---|---|---|---|
| DeepEval | Offline (Layer 1) | Strong (Ragas built-in) | OSS | Excellent (pytest-style) | Teams that want eval as Python tests |
| Ragas | Offline (Layer 1) | Native | OSS | Good | RAG-heavy products |
| Promptfoo | Offline (Layer 1) | Good | OSS (MIT) | Excellent (YAML, CI) | Prompt regression and red-team |
| LangSmith | Online (Layer 2) | Good | Hosted; self-host on Enterprise | Via SDK | LangChain-native stacks |
| Braintrust | Online (Layer 2) | Good | Hosted; self-hosted data plane | Excellent | Multi-model agentic systems |
| Arize Phoenix | Online (Layer 2) | Good | Self-host (ELv2) / hosted | Good | OpenTelemetry-native, self-hosted |
| Langfuse | Online (Layer 2) | Good | OSS (MIT) / hosted | Good | OSS-first, self-host friendly |
| Opik | Online (Layer 2) | Good | OSS / hosted | Good | Comet-aligned shops |
| MLflow (GenAI eval) | Enterprise | Good | OSS / Databricks | Excellent | Existing MLflow / Databricks users |
| HELM / lm-eval-harness | Research | N/A | OSS | N/A | Base-model selection only |
Choosing Your Stack: A Pragmatic Decision Tree
There is no single right stack. Here is how we usually shape recommendations.
- Pure LLM app, small team, fast iteration: Promptfoo for prompt regression in CI, DeepEval for Python-test-style eval in the same repo. No hosted platform until traffic justifies it.
- RAG-heavy product: Ragas for the metric set, plus a hosted tracer (Phoenix or Langfuse if self-hosting matters; Braintrust or LangSmith if it does not).
- Regulated or enterprise stack: MLflow GenAI evaluation if you are on Databricks; Vertex AI Gen AI Evaluation or the Foundry SDK if the rest of the platform is on GCP or Azure. Compliance posture beats marginal feature gaps.
- Multi-model agentic systems: Braintrust or LangSmith for tracing depth and custom scorer ergonomics. Pair with DeepEval or a thin custom harness for the offline regression layer.
On build versus buy: if you can articulate a two-year horizon in which an internal eval harness gives you specific leverage (proprietary scoring, data residency, integrations no vendor will build), build. Otherwise, buy. Vendor velocity in this space outpaces internal projects, and the cost of building stalls the actual product work.
If you are building an LLM-powered product and want help designing the evaluation stack and the operational discipline around it, our AI consulting team has shipped this for clients; evaluation suites are a standard part of our LLM development services.
