We built a feature, measured it, and shipped it switched off. A client should expect the same from anyone they hire to build AI.
The feature was a precision floor for AgentExperience.NET, our open-source library that gives agents a memory of their own verified runs. The floor is a threshold: a lesson only reaches the agent's context if it matches the current situation strongly enough. Weak matches stay out. Sensible on paper. In an earlier experiment, wrong lessons had measurably hurt, doing worse than no memory at all, so filtering out the weak ones looked like the obvious next step.
The rule came before the numbers
Before running anything, we wrote down how the threshold would be chosen. Sweep a fixed set of candidate values. Keep the ones that hold false positives under a stated ceiling on both storage backends and inject nothing unexpected on a set of deliberate distractor queries. Of those, take the one with the highest recall. If none qualifies, the floor stays off and has no default.
That last clause is the whole discipline. The rule had to say what would happen if the answer was no, and it had to say it before anyone had seen a table.
No value qualified
Then we ran the sweep. No value passed. Every threshold that removed noise also removed the right lesson for long, realistic requests, where a handful of matching words is a small share of the text. The remaining noise did not move as the floor went up, because the distractors that survived contained every word of a short query, and no coverage threshold can tell those apart from the right answer. The floor removed the signal before it removed the noise.
So it ships off, with the table that says why and the rule we fixed in advance, so nobody can accuse us of moving the goalposts once the result came in. Anyone who wants the floor can turn it on and see exactly what they are trading.
Why we are telling a business audience this
This is a story about a threshold in a retrieval system, which is not a thing most buyers of AI will ever touch. But the shape of the story is one every buyer should recognise, because it is the shape most pilots are missing.
Decide how you will judge a feature before you build it. Afterwards, every threshold looks reasonable and every vendor's demo looks like success. The only defence is a rule written while you still do not know the answer. We have made the opposite mistake in public: an engineer here once documented a benchmark result and wrote a convincing explanation for it before noticing the error margin was as wide as the effect. A good story is the most dangerous thing you can have next to an unexamined number.
A written negative result is worth more than an unwritten positive one. The next person to look at the precision floor starts from a table and a rule, not from hope. They will not spend a week rebuilding it on the same intuition we had, because the intuition is recorded alongside the reason it was wrong.
Shipping a measured "no" is a deliverable. It saved the feature's future users from a default that would have hurt them, silently, on exactly the long realistic requests where they most needed the right lesson. We hold the same view about builds: sometimes the eval kills the build, and saying so early is the most valuable thing a firm can do for a client.
The review that keeps the measurement honest
A measurement is only as trustworthy as the review of the code that produces it. The precision floor was one of ten pieces of work in the latest round on the library, each reviewed independently of the person who built it, and each review found something real: a measurement that treated its two groups differently, a lock that reached across tenants, erased data that survived in a search index. None of those would have shown up in a demo. All of them would have changed what the numbers meant.
That is the standard we hold ourselves to, and the one we recommend clients hold their vendors to. Ask to see the rule that was fixed before the pilot ran. Ask what the pilot found that went against it. A vendor with nothing in that column has either been very lucky or has not been looking.
If you want your current AI pilot held to that standard, start with a free AI readiness assessment. We will tell you what the check should be, what the rule for reading it should say, and, if the answer turns out to be no, we will put that in writing too.