Whether you're about to ship an AI agent your team built or sign off on one a vendor built, you need a way to answer one question: will this hold up in production, or just in the demo? Agents fail less from model limitations than from the platform around them lacking reliability — so the evaluation is mostly about that platform. Here's the checklist we run, and that you can ask any vendor to walk you through.
1. Is there an eval harness — and does it gate releases?
Ask to see it. Not "we tested it," but the actual dataset of real questions, real answers, and explicit thresholds, with a blocking set that stops a release when it fails. If the answer is "it looked good in testing," there is no eval, and the cheapest model that passes was never actually chosen — the vendor's favorite was. No green eval, no ship.
2. Can you trace a single request end to end?
Open one real (failed) interaction and follow it. A multi-agent system that returns a confident wrong answer while every component "succeeded" is the normal failure mode, and you can only debug it with a trace — typed spans, shared IDs, the reasoning attached — not a flat log. If nobody can show you why a request did what it did, nobody can operate it once it is live.
3. Is there a verifier between the model and anything irreversible?
For any action the agent can't undo — sending, paying, booking, deleting — there should be a check before it commits: a verifier gate, a human checkpoint, or both. "The model is usually right" is not a safety mechanism. Ask what happens on the wrong-but-confident output, four steps deep.
4. Is there a permission map?
Every agent that calls tools needs an explicit map of what it can read, write, mutate, escalate, and never touch — the boundary drawn before the first tool call, not bolted on after an incident. If the agent "can do anything the API allows," that's not a permission model, that's the incident waiting to happen.
5. Does it know when it's wrong?
A system that only has one mode — answer — will fabricate confidently when it should refuse or escalate. Refusal and escalation behavior is something you design and measure, not hope for. Check that the eval scores it.
6. What does it cost under real load, and is that measured?
Demo traffic is forgiving; production traffic compounds. Ask for p95 latency and dollars-per-request under realistic load, and whether the loop is cache-aware. A system that's fine in the demo can be slow and ruinously expensive at scale if nobody budgeted it.
7. Is there a runbook and a rollback?
When it drifts or breaks at 3 a.m. — and it will — is there an actual operating document: alerts, degradation modes, when to wake a human, how to roll back? "We'll monitor it" is not a runbook.
How to use this
For a build you own: this is the definition of done. For a vendor you're evaluating: ask them to walk your technical team through all seven on a past project. A genuine firm can. A demo-driven one will pivot to the roadmap. The gap between the two answers is exactly the gap between a system and a slide.
Evaluating an AI agent isn't about how impressive it looks — it's about whether the seven things above exist. They're the difference between "it works" and "it survives." If you want a second set of eyes on a system before you ship or buy it, that's literally our audit.