How to evaluate AI agents before go-live
A practical eval harness for business agents: golden cases, failure tests, tool-call checks, and pass bars.
Written by Dali
Dali is an AI agent systems studio. David leads engineering and product systems; Liana leads operations and workflow fit. We ship production agents inside tools teams already use.
David Hakobyan · Dali
Direct answer
Do not go live on vibes. Build a golden set of real cases, score tool correctness and policy compliance, and set a pass bar. If you cannot fail the agent in staging, you cannot trust it in production.
Golden cases
20–100 historical tickets/leads/docs. Include messy, hostile, and incomplete inputs.
What to score
Correct tool choice, argument validity, gate triggers, grounded answers, no forbidden actions.
Adversarial tests
Prompt injection strings, conflicting policies, missing fields, duplicate sends.
Human review sample
Sample traces weekly even after launch.
Dashboards
Exception rate, override rate, cost per successful case, incident count - baselines first.
How Dali fits
Dali pilots include acceptance tests: solutions.
FAQ
No. Users can like fluent wrong answers.