There are two things people call an "AI audit," and they could not be more different. One asks where should we apply AI — that's an opportunity audit, and we wrote about it in "The audit is the product." This piece is about the other kind: the AI system audit. You already built the thing. It's live, or nearly live, or you inherited it, or it just fell over in production. Now someone needs to tell you whether it's actually fit to run.
That's a different job with a different smell. An opportunity audit produces a roadmap. A system audit produces a verdict, a list of what will break, and the order to fix it in. If you're staring at a working demo and wondering whether you can trust it with real customers and real money, this is the audit you want.
First, know which audit you're buying
A system audit assumes the system exists. We're not scoring your strategy. We're reading your code, your traces, your prompts, your retrieval config, your access rules, and your bill. The output is boring on purpose: here's what's solid, here's what's a latent incident, here's what you're paying that you didn't know about.
Get one at four moments. Pre-launch, before you point it at customers. Inherited, when a system lands on your desk with no author around to answer for it. Post-incident, when it did something wrong and you need to know if that was a one-off or a category. And before scaling, when 100 users becomes 10,000 and every quiet flaw gets a megaphone.
Evaluation and quality: does it ship on vibes?
This is the first thing we look for and the most common thing that's missing. Ask the team one question — "how do you know a change made it better?" — and the answer tells you everything. If it's "we tried a few prompts and it looked good," you're shipping on vibes.
What we look for: an eval harness that runs on every change. A ground-truth set that reflects real, ugly production inputs — not twelve happy-path examples someone wrote on a Friday. Regression tests, so fixing one case doesn't silently break five others. A quality metric that a human actually agreed with at some point.
Common failure: no eval at all, and a team that improves the system by feel. The second most common: an eval that exists but was written once, never updated, and no longer resembles what users actually send. A stale eval is worse than none, because it manufactures confidence.
Security and access control: whose data can it see?
AI systems love to leak, and they leak in ways traditional apps don't. The classic hole: retrieval and tools run with more permission than the user asking. Someone queries the assistant, and it happily pulls a document they'd never be allowed to open in the real app.
What we look for: does retrieval respect the user's actual permissions, enforced at query time, not filtered after the fact in the prompt? Are data boundaries between tenants real, or one boolean away from collapse? Is PII handled deliberately — redacted, scoped, logged with care — or is it flowing into prompts and third-party model logs by accident? And prompt injection: if your system reads untrusted text (emails, web pages, uploaded files), assume that text will try to hijack your tools. We check whether there's any defense, or just optimism.
Common failure: permissions enforced in the UI but not in the retrieval layer. The model becomes a confused deputy with god-mode access.
Cost and efficiency: the run-cost nobody measured
Almost every system we audit is paying more than it needs to, and most teams can't tell us their cost per request without going to look it up. That's the tell.
What we look for: token economics per request — input, output, and the retrieval/tool overhead nobody counts. Context bloat, where the prompt has quietly grown to stuff the whole knowledge base into every call. Caching — prompt caching for stable prefixes is often free money left on the floor. And model routing: are you running your most expensive model on tasks a cheaper one handles fine?
Common failure: a fixed, giant system prompt sent on every request with no caching, plus over-retrieval that triples token count for no measurable quality gain. The fix often cuts cost by half without touching quality.
Observability and tracing: can you see one request?
Pick a single real request that went wrong yesterday. Can you pull up exactly what happened — the prompt, the retrieved chunks, the tool calls, the model output, the latency of each hop? If the answer is "we'd have to add logging," you are flying blind and you'll debug the next incident the same way: by guessing.
What we look for: end-to-end tracing on every request, not just errors. Spans across retrieval, model calls, and tools. Quality and drift monitoring in production — not just uptime, but whether outputs are getting worse as inputs shift. The modern tracing and eval tooling makes this cheap now; there's no excuse for a black box.
Common failure: logs that capture the final output but not the intermediate steps, so every debugging session is an archaeology dig.
Reliability and failure modes: what happens when it breaks?
It will break. The model API will time out, a tool call will return garbage, retrieval will come back empty. The question is whether your system degrades gracefully or takes the whole request down with it.
What we look for: timeouts on every external call, with sane defaults. Fallbacks — a cheaper model, a cached answer, an honest "I can't do that right now" instead of a hang or a hallucinated guess. Handling for malformed tool calls and invalid model output, because both happen constantly at scale. Retries that don't stampede. We red-team the unhappy paths on purpose, because that's where production actually lives.
Common failure: a happy-path system with no fallbacks, where one slow dependency turns into cascading timeouts and a dead user experience.
Data and retrieval quality: the RAG-specific trap
If it's a RAG system, retrieval quality is the ceiling on everything. A perfect model reasoning over the wrong three chunks gives you a confident wrong answer.
What we look for: chunking that respects the document's real structure instead of blind fixed-size splits. Retrieval precision — are the top results actually relevant, measured, not assumed? Freshness — when the source changes, how long until the index knows? We test retrieval in isolation, because a retrieval bug and a generation bug look identical from the outside and get fixed in completely different places.
Common failure: nobody ever measured retrieval on its own, so a precision problem gets misdiagnosed as a prompt problem and the team spends a month tuning the wrong layer.
What you get
Zoff.tech runs a fixed-scope production AI system audit across exactly these dimensions — evaluation, security and access, cost, observability, reliability, and retrieval. You get a written verdict, a prioritized list of what will break and what it's costing you, and the concrete fixes in the order that matters. Not a deck you could hand to your competitor with the logos swapped. A read of your system, and a straight answer about whether it's ready. Get one before launch, or after the incident that made you go looking for this article.