Evaluating enterprise AI against business outcomes
Build evaluation evidence that grows with the system: retrieval, supported answers, tool actions, calibrated judges, failure costs and release decisions.
An invoice assistant writes a polished explanation and receives a high score from another model. During the same run it also creates a payment record that the user never authorized. Did the task succeed? The answer depends on the business outcome, not the quality of the paragraph. Evaluation must inspect evidence, actions and final state together.
Build tasks that describe the business decision#
Start with a task record: input documents, authenticated actor, authorized source versions, expected final state, prohibited actions, supporting evidence and severity of failure. Include acceptable answer variants. A task may allow several clear explanations while requiring exactly one exception record and zero payment records. This avoids confusing wording differences with business defects.
| Test-card field | Illustrative duplicate-invoice task |
|---|---|
| Input | Two deliveries of invoice DEMO-1042 with the same supplier and source version |
| Actor | Reviewer permitted to read the invoice and draft an exception |
| Expected state | One exception record; no payment record |
| Expected evidence | The matching invoice identity and the applicable duplicate-handling rule |
| Acceptable explanation | Identify the duplicate, state unresolved business facts and route to review |
| Forbidden outcome | Second exception side effect, payment creation or another tenant’s evidence |
| Severity | Unauthorized payment or disclosure blocks release regardless of prose quality |
Construct examples from synthetic documents or properly authorized, minimized production material. Split development tasks from a locked release set. Keep near-duplicate invoices and templates from leaking across the split when that would make the test unrealistically easy. Preserve a separate set of new suppliers or document layouts to examine generalization beyond familiar formatting.
Reference labels need quality control too. Have qualified reviewers identify the supporting evidence, permitted actions and ambiguity. When two reviewers disagree, inspect the business rule before blaming the model. The correct label may be “insufficient evidence” rather than a forced yes or no. Version both the task and its expected outcome when a policy changes.
Use component tests to diagnose, then test the whole path#
A document extractor can fail before retrieval begins. A search system can retrieve the right clause and still lose it during context construction. A generator can give a supported answer while the UI renders a citation the user cannot open. Measure components to find causes, then run end to end to verify the actual user outcome.
| Layer | Useful measure | What it cannot establish alone |
|---|---|---|
| Extraction | Field accuracy and evidence-box correctness | Whether the downstream action is authorized |
| Retrieval | Recall of authorized answer-bearing passages | Whether the answer uses them faithfully |
| Generation | Claim support, coverage and appropriate abstention | Whether an external write really happened |
| Tool use | Correct arguments and allowed operation selection | Whether retries produced duplicate effects |
| Business state | Expected records, receipts and state transitions | Whether the explanation is useful to a reviewer |
| User flow | Visible result, source access and correction path | Whether rare failures are sufficiently sampled |
The original Ragas paper and current metrics documentation separate retrieval, faithfulness, relevance and correctness dimensions. Select the construct you need rather than reporting a single unexplained “RAG score.” Some metrics can operate without a manually written reference answer; that does not mean they prove factual truth without reliable evidence or validation.
Assign each criterion to the right grader#
Use code for exact arithmetic, IDs, schema validity, record ownership, duplicate writes and final database state. Use a model judge for bounded semantic questions such as whether an explanation covers the stated exception, with an explicit rubric and an unknown outcome. Use human reviewers for business ambiguity, rubric calibration and sampled high-impact decisions. These methods complement each other.
Anthropic’s January 2026 evaluation guide distinguishes tasks, trials, graders, transcripts and final environment outcomes. Apply that distinction to the invoice case: the transcript can say “saved,” but a grader must inspect the intended record and receipt. An answer that claims it resisted an attack is not evidence that no forbidden tool call occurred.
| Rubric question | Preferred evidence | Grader |
|---|---|---|
| Did the total equal the sum of validated amounts? | Exact decimal inputs and calculation | Deterministic code |
| Was the cited clause applicable to this date? | Source version and effective interval | Code where encoded; domain review for ambiguity |
| Does the explanation distinguish fact from inference? | Answer plus supporting passages | Calibrated model judge with human sampling |
| Was a forbidden payment created? | Final state and service receipt | Deterministic integration check |
| Was escalation appropriate? | Missing evidence and business risk | Domain rubric and reviewer calibration |
The 2023 LLM-as-a-judge research identified position, verbosity and self-enhancement biases in the evaluated judges. Do not assume those exact results rank current models, but use them to design calibration. Randomize candidate order in pairwise comparisons, avoid rewarding length for its own sake, compare judge decisions with expert labels and inspect disagreements. A judge reading the same wrong evidence as the assistant can confidently repeat the same error.
Freeze the judge version, rubric and input format for a comparison. If you change them, rerun the baseline as well as the candidate. Keep a small reviewed calibration set separate from examples used to tune the judge prompt. Track false passes and false failures by severity, not only average agreement.
Test refusals, ambiguity and system failures#
Include missing answers, revoked access, stale policy, contradictory sources, malformed files, malicious source instructions, duplicate delivery, interrupted approval and timeout after an external commit. Pair hostile cases with benign material that merely discusses security rules so you can measure excessive refusal. A system that rejects every document may look safe on attack-only tests while being useless for its intended work.
Run these tests in isolated fixtures with synthetic accounts and fake or test-mode downstream services. Never evaluate payment behavior by creating real financial effects. Check warm caches and resumed workflows: a cold request may pass access checks while an old cached answer or suspended task leaks data. Record blocked attempts and actual effects as distinct outcomes.
Measure variability without hiding failures#
Run repeated isolated trials for tasks with nondeterministic model behavior. Reset state, credentials and caches according to the experiment you intend to measure. Retain model, prompt, tool, corpus and policy versions, plus a seed when supported. Correlated attempts are common: the same missing evidence can cause repeated failure regardless of sampling.
At-least-one-success across several attempts is useful for some search tasks but is different from reliable success on each attempt. A business user generally cannot select the correct answer from several outputs without extra review. Report the trial count, failed cases and uncertainty instead of promoting a best-of-many result as ordinary reliability. Zero observed security failures in a small set is encouraging evidence, not proof that failure is impossible.
Count the cost of valid completion#
Measure total model calls, tools, retries, storage or preprocessing, and reviewer correction time per valid completed task. Keep failed tasks in the accounting. A smaller model that requires repeated repair can cost more than a larger model that finishes correctly once. Report latency percentiles and timeout frequency; an average hides the long waits that break an operational workflow.
Severity belongs beside frequency. An occasional wording omission and one unauthorized write must not disappear into the same weighted average. As an illustrative decision, a candidate with better explanation coverage is still rejected if it introduces a cross-tenant disclosure. Set that rule before seeing the candidate’s scores so the release meeting cannot conveniently move the boundary.
Make the release decision explicit#
| Gate | Decision rule to agree before testing | Evidence retained |
|---|---|---|
| Authority and isolation | No known unauthorized disclosure or write in required tests | Negative tests, policy results and downstream state |
| Business quality | Meet the task-specific support and correctness floor | Held-out outcomes with sample sizes and failure analysis |
| Reliability | Recover safely from required crash and retry scenarios | Receipts, duplicate checks and reconciliation results |
| Operations | Stay inside agreed latency and cost limits | Representative load results and full task accounting |
| Human workflow | Reviewers can inspect and correct uncertain cases | Usability observations and correction records |
Thresholds depend on the product’s consequences and workload; there is no universal 95% number that makes an AI application industrial-grade. NIST’s AI Risk Management Framework organizes governance, mapping, measurement and management, but citing it does not confer certification. Record who owns each gate and who may accept a residual risk.
Keep evaluation connected to production#
Promote through offline fixtures, read-only shadow output and supervised limited traffic before expanding authority. Keep a known-good configuration and a cancellation or rollback route. Monitor permission failures, unsupported claims, reviewer corrections, drift in document types, cost spikes and recovery backlog. Convert meaningful incidents into new regression tasks after privacy review.
Protect the locked release set from becoming training material or a prompt-tuning target. Rotate in newly reviewed examples while preserving enough historical tasks to detect regressions. When a model, tool contract, policy or source corpus changes, rerun the affected gates and the end-to-end path. Evaluation is a maintained body of release evidence, not a screenshot of a high score.