TensorPrimitives made my code 2.5× faster on one operation and 1.5× slower on the other

I had two vectorized primitives hand-written with Vector<float> and the same suspicion about both: that the standard library version would be better. I replaced both with System.Numerics.Tensors and measured. One won by 2.55×. The other lost by 1.47×.

This isn't about implementation quality — same team, same kernel, same version. It's structural, and once you see the pattern you can predict it in advance. But you cannot predict it by reading the API.

Measurement environment: Apple M3 Pro (11 cores), macOS 26.3, .NET 10.0.10, Arm64 RyuJIT armv8.0-a, BenchmarkDotNet 0.15.8, default job. Vector<float> is 4 wide here.

The two hand-written SIMD primitives in C#

The library — a from-scratch feed-forward neural network, which here is just a test bench — rests almost all of its compute time on two operations:

• Dot(a, b) — dot product, written with two accumulators to break the dependency chain the summation imposes.
• AddScaled(dest, src, scale) — dest += src * scale, an element-wise multiply-accumulate.

They look like siblings: walk two float arrays, multiply, add. The obvious substitution was TensorPrimitives.Dot and TensorPrimitives.MultiplyAdd. The equally obvious working hypothesis: the library wins both.

Result 1 — Dot loses

The interesting row isn't the first one but the last, and the key is comparing it against the column nobody looks at.

TensorPrimitives.Dot at 4096 elements takes 734 ns. The single-accumulator version takes 723 ns. Under 2 % apart. The library kernel is performing exactly like my naive single-chain version, while my two-accumulator version finishes in 500 ns.

The mechanism: a dot product ends in a reduction. All the partial products have to collapse into one scalar, and if they accumulate into a single register then each add depends on the previous result. That's a serial chain, and its length — not the SIMD width, not the number of multiplies — sets the time. The second accumulator exists precisely to split it into two independent chains the execution units can overlap. The TensorPrimitives kernel drags a single chain through the reduction, so it pays the same price.

At length 8 it loses for a more prosaic reason: it's a real call, while my version gets inlined. With 8 floats, the call cost is comparable to the work.

Result 2 — AddScaled wins

The same call-cost penalty at length 8, and from there a win that grows past 2× and levels off.

The structural reason — and this is the heart of it: AddScaled is a pure streaming operation. One output value per input value, no reduction. There is no dependency chain to carry from one iteration to the next, so the kernel can unroll the loop as far as it likes and keep every execution unit fed. My hand-written loop doesn't unroll.

A second factor compounds it: MultiplyAdd emits a real fused multiply-add, while dest[i] += src[i] * scale compiles to a separate multiply and add.

Reduction versus streaming. That's the whole criterion.

Result 3 — does it show up in training?

A microbenchmark proves nothing until you weigh it against everything it shares work with. Winning 344 ns in an operation that runs a hundred times a second changes nothing.

So I measured a full mini-batch step — forward, backward, and weight update — on a 784 → 128 tanh → 10 softmax network with batch size 32:

A third off the full step. But the row that makes the experiment trustworthy is the middle one.

AddScaled does not appear in the forward pass. By construction, that number had to stay put with and without the change. If those two values had diverged, the right conclusion wouldn't have been "the win is even bigger" but "something moved other than what I think I changed" — a different build, a different machine state, a badly isolated benchmark.

They differ by 0.3 %, inside the noise. That places the entire 207 µs difference in the backward pass, exactly where the primitive lives. A control row costs one line of the table and turns a correlation into an attribution.

The lesson: reduction versus streaming

Same library, same version, two operations that look like siblings in the source, and opposite answers. The difference isn't how much effort went into each kernel: it's whether the operation ends in a reduction.

That yields a usable heuristic:

• Streaming operations — element-wise maps, axpy, scaling, vector adds — tend to favor the library kernel, which unrolls better than you're going to by hand.
• Reductions — dot products, norms, sums, maxima — depend on how many independent accumulation chains the implementation maintains. A hand-written version with several accumulators can win there, and it's worth checking.
• At small lengths, call cost dominates and the inlined version almost always wins. If your typical size is a few dozen elements, measure before replacing anything.

And above all: "use the optimized library function" is a hypothesis, not a conclusion. It's a good hypothesis — it was right about one of the two — but neither decision was predictable from reading the API, and the only way to find out which was which was to measure both.

The usual platform caveat: every figure here is ARM64 — M3 Pro, NEON, Vector<float> 4 wide — on .NET 10.0.10. On x86 with AVX2 or AVX-512, where vectors are 8 or 16 elements, the ratios may change; the structural argument about reductions holds, the specific numbers don't.

Worth adding that this library is not fast in absolute terms: it's single-threaded, has no real batching, and no blocked GEMM. It works as an honest test bench for these primitives, not as a performance reference.

The code

Everything above is reproducible: neural-network-csharp. The benchmarks are in bench/, with the results discussed in bench/README.md.