ztzoff.tech

Nov 3, 2026

Weights, documents or experience: where each kind of knowledge belongs in an agent

Fine-tuning for habits, RAG for what someone wrote down, and a verified memory for what the agent learned last week. The third layer augments the model. It does not replace training.

When an agent keeps getting something wrong, the fix depends on what kind of knowledge is missing. We find it useful to sort knowledge into three places, because each place has a different tool, a different cost, and a different way of going stale.

In the weights

Style, output format, how to call your tools, the vocabulary of your domain: anything that should hold in every run, for every user, regardless of the task. This belongs in the model itself, through fine-tuning, LoRA or reinforcement learning.

It costs a training run and model access, which rules it out for most hosted frontier models. It is also hard to take back. If one example taught the model a bad habit, you cannot delete that example; you retrain. We have argued before that this is the wrong tool for business facts, and nothing here changes that. Weights are for behaviour.

In documents

APIs, policies, manuals, contracts, the price list: things a person wrote down. This belongs in retrieval. The model reads the relevant passage at query time, can cite it, and the knowledge is only as stale as the document. Access control runs through your existing permissions. This is the layer most teams already have, and for written knowledge it is the right one.

Its limit is in its definition. Retrieval returns what someone wrote. It does not know what your agent tried yesterday, because nobody wrote that down.

In experience

This test fails with this error in this repository, and this command fixed it. That dependency upgrade breaks the health check, and the fix is a config line nobody documented. Things that happened, in your systems, that change weekly, and that no document contains because the only witness was the agent.

This is the layer most agents lack. It must be earned from runs rather than authored, and to be worth anything it has to be trustworthy. That is what we built AgentExperience.NET to hold, and what makes experience different from the other two layers:

  • It is earned, not authored. A lesson exists only if a check outside the model passed: a test run, an exit code, a person's sign-off. The model cannot vouch for itself. A run that failed is quarantined and never reused.
  • It is keyed on the situation. Lessons are indexed by how a tool failed, not by the request text. What predicts whether a lesson applies is the failure the agent is in. We described this in why agents repeat the same mistake.
  • It arrives at the moment of failure. A bounded hint is added to the agent's next model call in the same run, labelled as untrusted reference material, not as an instruction.
  • It is reversible. Delete one record, or erase one user, and the effect is immediate and checkable. You cannot say that about a weight update.
  • It is measured. A random holdout withholds lessons from a share of runs, and the result is helps, hurts, or no evidence yet, with the interval. Every pilot should have one.

The layers combine

A tuned model that knows your tools. Documents that state your policies. A memory that knows what broke yesterday. These are not competitors, and the question "RAG or fine-tuning or memory" is a sign the problem has not been sorted yet. The useful question is: what does the agent get wrong, and is the missing knowledge a habit, a fact someone wrote, or a thing that happened?

We are not claiming memory beats a fine-tune. We are claiming it covers something the other two cannot: learning from a single run, immediately, on a model you cannot train, in a form you can delete.

The honest status of the third layer

The loop and the measurement are built and reviewed, and the evidence is mixed. On a synthetic task, lessons help, with a narrow result: one run per model, 12 instances, the strategy reaches the agent verbatim. On realistic tasks, the only evaluable run showed no overall benefit against no memory (p = 0.50), although the content of a lesson did matter, with real lessons beating a placebo block on 7 of 8 differing services (p = 0.035). Lessons from the wrong breakage did worse than no memory at all. A transfer experiment showed no demonstrated benefit (p = 0.0625). The run on real development work is pre-registered and not yet switched on.

We report that because the point of the third layer is that it can be measured, and a layer that can be measured will sometimes measure badly. Weights and documents rarely get that scrutiny. They should.

What to do with this

If your agent gets the format wrong, look at the weights or the prompt. If it gets facts wrong, look at the documents and the retrieval in front of them. If it rediscovers the same fix every Tuesday, that is experience, and the first step is deciding what check your own code can run to say a run worked. Without that check there is no third layer, only a notebook of the model's opinions.

Sorting a client's failures into these three places, and naming the check, is what we do in a free AI readiness assessment. It usually takes an afternoon, and it ends most arguments about which tool to buy.

The library is open source and in preview. The Microsoft Agent Framework adapter is on NuGet; the command-line tool and MCP server for other agents ship from the next preview, and the repository has every experiment, including the negative ones.

Book a discovery call