The promise of a self-improving model is seductive: it generates harder problems, solves them, learns from its own solutions, and repeats. No human labeling, effectively infinite data, gains that compound on themselves. The failure mode is just as clean. A model that generates its own training data and judges it will confidently reinforce its own mistakes. The loop doesn't improve — it drifts, quickly, in a direction nobody chose.
The difference between those two outcomes is one component: a verifier the model can't talk its way past.
Generation was never the bottleneck
The Evol-Instruct line of work — MMEvol and its relatives — evolves a small seed set of instructions into a larger, harder, more diverse dataset. It works, and it's genuinely useful. But notice what it produces: more data. And more data was never the constraint. The constraint is knowing which of it is correct. A pile of evolved problems with confidently-wrong answers doesn't teach a model anything except to be confidently wrong at scale.
The verifier is what makes the loop work
The versions that actually improve a model add an external signal of correctness. WizardMath uses reinforcement learning from Evol-Instruct feedback — rewarding the model against a signal for whether the reasoning was right, not whether the model liked its own answer. The reported result is striking: a 70B WizardMath model outperforming GPT-3.5-Turbo, Claude 2, and Gemini Pro on math. Not because it generated more, but because something other than the model decided what counted.
The structural point survives even if you ignore the leaderboard: generation and verification have to be different things. The moment the generator is also the judge, the gate is decorative — it will always approve, because it produced the candidate in the first place. Self-grading is not a quality control; it's a confidence amplifier.
The production lesson, wearing a training-loop costume
This is the same shape we keep arriving at, just at training time instead of serving time. AI slop is what shipping without a gate looks like; a self-improving loop without an independent verifier is the same failure with a feedback cycle bolted on — it gets more confident, not more correct. The strongest agent systems put the same discipline in the runtime: a verifier checking the action before it's allowed to count.
Whether it's training data or live output, the rule doesn't change: the model produces infinite candidates, and the value lives entirely in the thing that decides which ones to keep. That decision is also what turns "we generated a lot" into "the cheapest model that passes" — an eval, not a vibe.
Generation is cheap and getting cheaper. Judgment is the scarce part — and it cannot come from the thing being judged. Build the verifier separately, or your self-improvement loop is just a faster way to memorize your own mistakes.