AI evals

A cousin of ML benchmarks.

At the Evaluating evals workshop at NeurIPS 2024, I decided I needed to understand the distinction between “benchmarks” and “evals” in ML/AI.

Benchmarks are standardized tests used to compare AI models on common tasks, while evals, as far as I can tell, are broader, often custom, evaluations used to judge whether a model is fit for a particular use case or deployment context.

That is to say, benchmarks in ML/AI are fixed datasets and protocols meant to provide “apples-to-apples” scores across models or model variants (e.g., MMLU, GSM8K, TruthfulQA for LLMs). They emphasize comparability and repeatability: same inputs, same metrics, so we can rank or track progress over time and publish leaderboards or marketing claims like “X% on benchmark Y.” Conceptually, benchmarks mostly answer “What can this model do in general, relative to others?” rather than “Is it good enough for my specific workflow?”

“Evals” (evaluations) are a broader category that focuses on how a system performs for a concrete task, product, or risk profile. In contemporary LLM practice, evals are often application-specific suites (e.g., for a customer-support bot, legal RAG system, or internal agent) that may mix automated metrics, human judgements, and logs from real users. They tend to be iterative and operational, used for error analysis, regression testing, safety checks, and release gating, rather than just for external comparison.

Benchmarks are typically public, static, and model-centric, while evals are often private, dynamic, and product- or context-centric (e.g., built from our own user data and goals). In practice, there is overlap—internal eval suites can double as “benchmarks” for marketing, and published benchmarks can be part of an eval pipeline—but the intent differs: comparison versus decision-making for a specific deployment.

More recently, a school of evals called EvalEval has arisen. EvalEval is a research coalition focused on meta-evaluation: understanding how benchmarks behave over time, how to design better evaluations, and how to document and standardize them (e.g., “evaluation cards,” harness tutorials). Their “evals” are not ad hoc; they are systematically designed, documented, and scientifically grounded studies of AI systems’ capabilities, risks, and broader societal impacts. They claim to be more like classical science than the benchmark world science: their approach treats AI models and their evaluations as observable phenomena that we can study in their own right, and which we analyze via RCTs, standard experimental methods, and statistical techniques, as I learned at NeurIPS 2025.

EvalEval also explicitly targets “broader impact evaluations” and decision-makers, meaning assessments of downstream risks, opportunities, and externalities, not just accuracy or utility for a single product. That adds a policy and governance dimension that goes beyond the earlier framing of evals as primarily for model and feature decisions within a single organization.

I am happy with this framing.