For a while my library's documentation stated something false, with a table beside it and a paragraph explaining why it was so. The paragraph was good. That was the problem.
Measurement environment: Apple M3 Pro (11 cores), macOS 26.3, .NET 10.0.10, Arm64 RyuJIT armv8.0-a, BenchmarkDotNet 0.15.8.
Vector<float>is 4 wide here.
The claim
My vectorized dot product uses two accumulators instead of one, to break the dependency chain the summation imposes and let the CPU overlap two independent chains. At every measured length that won time… except one.
At length 8, the documentation said, the second accumulator cost 17 %: 1.54 ns against 1.27 ns for the single-accumulator version. A 0.83×. A negative result, honestly published, with its exception carefully bounded.
The explanation, which was too good
And underneath the number sat a perfectly reasonable mechanism, already written:
Eight floats are exactly two 4-wide vectors. The two-accumulator loop processes two vectors per iteration, so it runs its body once and falls straight into the tail. It pays the whole setup — initializing two accumulator registers, the final horizontal sum of both — and recovers nothing, because there is no second iteration for the pipeline to overlap with. At the exact point where unrolling stops amortizing, the fixed cost shows in full.
All of that is true as a mechanism. It fit the number. And that is why nobody — me — looked at the number again.
The number was noise: three BenchmarkDotNet iterations
Those rows had been measured with BenchmarkDotNet's --job short, which is three iterations. When I came back to the table and looked at the column I had been ignoring, the error margin on those rows was ±0.26 ns.
The effect they claimed to describe was 0.27 ns — the difference between 1.54 and 1.27.
The error margin was as wide as the effect. The table wasn't saying "the second accumulator costs 17 %"; it was saying "with this configuration I cannot tell these two versions apart." I read the first thing and wrote a paragraph explaining it.
Re-measured with the default job, the result inverts: the second accumulator wins by 1.23× — 1.03 ns against 1.27 ns — just like at every other length. There was no exception. There never had been.
| Length 8 | 1 accumulator | 2 accumulators | |
|---|---|---|---|
--job short (3 iterations, ±0.26 ns) | 1.27 ns | 1.54 ns | 0.83× — inside the noise |
| Default job | 1.27 ns | 1.03 ns | 1.23× |
The two lessons about measuring with BenchmarkDotNet
The first is obvious and the less interesting: a three-iteration job is for triage, not for conclusions. --job short exists to tell you whether a change moves something by an order of magnitude while you iterate. The moment a number is going into documentation, it gets re-measured with the default job. And you read the error margin every time, not only when the result surprises you.
The second is the one that actually cost me: a convincing explanation is not evidence.
The story about loop setup and tail iterations was persuasive, technically correct as a mechanism, and accurately described a real phenomenon that could have been happening. That is exactly why it worked so well as armor. A loose, odd number invites you to repeat the measurement. A number with a plausible mechanism written under it stops looking like a doubtful data point and starts looking like an understood fact — and understood facts don't get asked for their evidence again.
I wrote the explanation after seeing the number, for that number. That is precisely the order in which you fool yourself. The question I failed to ask — and now try to always ask — is: if the number were the other way around, would I have an explanation for that one too? With the two-accumulator loop I did, and it was the one already written across every other row of the table.
The optimistic counterexample
I hit the symmetric case in the same project, and it ends well.
TensorPrimitives.SoftMax computes exp(z)/Σexp(z) literally, without subtracting the maximum of the logits before exponentiating. That's the standard numerical stabilization, and without it the function returns NaN for logits above ~88, which is where exp overflows in single precision.
Two unit tests that already existed caught it on the first run after the substitution. Total cost of the bug: seconds.
That's the difference between the two failures in this post. The softmax one had a check waiting for it, so it lasted as long as the test suite took to run. The benchmark one had none — a well-written explanation is not a check, however much it feels like one from the inside — and it lasted until someone thought to re-read a column of error margins.
The code
Everything above is reproducible: neural-network-csharp. The benchmarks are in bench/, with the results discussed in bench/README.md.