AI Evaluation Must Follow the Workflow

The Comfort of a Model Score

Accuracy benchmarks help compare models under controlled conditions. Production systems are not controlled conditions. They retrieve changing information, call tools, inherit permissions and hand results to people who interpret them differently.

A strong model inside a weak workflow can still create unreliable outcomes. A modest model inside a carefully bounded workflow may perform dependable useful work.

Evaluate the Whole Chain

Testing should follow the path a real request takes:

• Was the request understood and routed correctly?
• Was relevant, current evidence retrieved?
• Were tools selected with valid arguments?
• Were permissions and approval rules enforced?
• Did the final action improve the intended business outcome?

Each stage needs its own observable failure categories. Otherwise every incident is labelled a model error and the real system weakness remains hidden.

Use Production Evidence Carefully

Offline test sets should be joined by sampled production traces, human review and outcome measures. Privacy controls and redaction remain essential, but evaluation that never sees real operating conditions will miss the edge cases created by them.

Teams should also track abstention and escalation. A system that recognises uncertainty can be more valuable than one that attempts every case.

Build a Workflow Scorecard

A useful scorecard separates model quality from system quality. It can track retrieval coverage, unsupported claims, valid tool calls, permission violations, reviewer corrections, successful escalations and the final outcome for the customer or operator.

Segment results by workflow and risk. An average score can hide excellent performance on routine cases and serious weakness on the small set that matters most. Review failed and narrowly successful cases together; near misses often reveal brittle assumptions before they become incidents.

Evaluation should also trigger decisions. Define what blocks a release, what reduces autonomy and what requires a new test case. A dashboard without consequences is observation, not governance.

Build Cases From Work, Not From Prompts

A realistic evaluation case contains the original request, available records, tool state, user permissions and the outcome the organisation considers acceptable. Testing only a prompt and response removes the conditions that make production difficult.

Cases should represent routine work, ambiguous work and high-consequence exceptions. Include stale documents, conflicting sources, unavailable tools and requests from users who lack permission. Preserve the expected escalation path as carefully as the expected answer.

The set should evolve from incidents, reviewer corrections and near misses. Synthetic cases are useful for coverage, but production evidence reveals the language, timing and organisational assumptions that designers did not predict. Each new case should identify which part of the workflow it is intended to challenge.

Make Evaluation Part of Release Engineering

Run the workflow suite when models, retrieval sources, prompts, tools or permissions change. Compare results by risk category and inspect regressions rather than relying on one aggregate score.

For consequential workflows, define a release gate and a rollback condition. Evaluation becomes valuable when it changes what the team is willing to ship, not when it merely produces an attractive dashboard after deployment.

Use Human Performance as Context, Not an Excuse

Comparing an AI workflow with perfect theoretical performance is useful for finding defects, but product decisions also need a realistic baseline. Measure how trained people handle the same cases, including disagreement, delay and escalation. This reveals where automation offers genuine improvement and where the task itself is ambiguous.

The baseline should not become permission to reproduce avoidable human errors at scale. Automation changes the distribution of harm: one weak instruction can affect thousands of cases consistently. Evaluation should therefore compare average quality, severe failures and the ability to detect uncertainty.

Where people and automation collaborate, evaluate the combined system. A mediocre draft may still save time if reviewers correct it reliably; a fluent draft may reduce quality if it encourages shallow review. The outcome belongs to the workflow, not to either participant in isolation.

Conclusion

Enterprise AI is a system of models, data, tools, permissions and people. Evaluation should use the same boundary and follow a request until its real-world consequence is known.

The question is not only whether the model answered well. It is whether the complete workflow behaved in a way the organisation can explain, improve and trust repeatedly—not just during a demonstration.