Most AI agent pilots we are asked to look at share one defect. Nobody can say whether the agent is helping.
There is a demo. There are anecdotes, usually good ones, from the people who wanted the pilot. There is a dashboard of how many times the agent ran. There is no comparison, because the agent was switched on for everyone at once, and the only "before" is what people remember. We have written about why hours saved is a number that lies. This essay is about the design that replaces it, and the reason it is cheaper than it sounds.
The design
Hold out a small, random share of runs. Those runs get the plain agent: no memory, no new prompt, no new tool, whatever it is you are testing. Everyone else gets the change.
Judge every run by a check that is not the model. Tests pass. The ticket closed without a reopen. A person approved the output. The model's own opinion of its work is not a check, for reasons we covered in a model can't grade its own homework.
After enough runs on each side, you have a difference between the two groups, with an interval, per model, and one of three verdicts: helps, hurts, or no evidence yet. The rule for reading that number was written down before the data came in.
We built this design into AgentExperience.NET, our open-source library for agent memory, so anyone can inspect it: a holdout is on by default, it compares verified outcomes, and it will not call a verdict before each arm has enough runs. The same design, with your checks and your runs, is what we set up for a client's agent.
Three things a holdout gives a leadership team that a demo never will
You find out when something hurts. In our own measurements on a realistic task, lessons taken from other breakages did worse than no memory at all: 0.303 of trials verified, against 0.455 without them. Without the holdout that would have looked like success, because the agent still finished its work most of the time. Nobody would have noticed that it finished less often.
You can switch things off with evidence. A lesson measured as harmful stops being shown automatically. The same logic applies to a prompt change, a new tool, or a new model: the thing that hurt gets turned off because a number said so, not because someone lost an argument.
You stop arguing. The number is the same whoever reads it. The vendor's champion and the vendor's sceptic see the same interval. When the rule for calling a result was fixed in advance, there is nothing left to negotiate after the fact, and the meeting about whether the pilot worked becomes a short one.
What it costs
Not much. A few percent of runs stay on the old path, which is a cost you are already paying for everyone today. The verifier is usually something you already have: your test suite, your approval step, your ticket system's closed-without-reopen state. What a holdout really costs is the willingness to find out, and that is the part most pilots skip.
Two things do take care. The holdout must be random, assigned by the system and not by a person choosing which runs look safe. And the groups must be treated identically in every way except the one being tested. An independent review of our own library caught a measurement that assigned its two groups differently, which would have quietly biased every result. That is the kind of error a holdout invites, and the kind that review exists to catch.
What our own holdout says so far
We hold ourselves to the design, so here are the numbers it has produced, none of them rounded up.
- On a synthetic task, lessons helped, narrowly: one run per model, 12 instances, and the strategy reaches the agent verbatim. It shows a model acts on an injected lesson, not that memory helps on real work.
- On realistic tasks, the only evaluable run showed no overall benefit against no memory (p = 0.50). Against a placebo block, real lessons used fewer tool calls on 7 of 8 differing services (p = 0.035). Every other run was inconclusive or stopped early.
- On CI triage, 1 of 7 correct with lessons and 3 of 9 without. The verdict is no evidence, for lack of runs.
- On real work, the run is pre-registered and not yet switched on. There is no result.
A library whose own measurement says "no overall benefit yet" is, we think, a better advertisement for the method than one that claims a win. It is the same honesty a client gets about their own system.
Where to start
If your agent is already live for everyone, you can still carve out a holdout: route a random share of new runs to the old path, start judging both with your existing check, and wait for the arms to fill. If the agent is not live yet, build the holdout before the launch, alongside the evaluation you should be running anyway.
Either way, the first step is naming the check. That is what we do in a free AI readiness assessment: identify the verifier you already have, size the holdout, and write down the rule for reading the result before anyone sees it.