A local optimization made my code 6× slower, and only the disassembly explained it

I vectorized the activation function of a dense layer and the layer got 6.3× slower. Not slightly slower: from 12.79 µs to 80 µs, with identical arithmetic, and with the vectorized part measuring twice as fast when I benchmarked it on its own. Nothing in the source code hinted at why. The explanation was in the disassembly, and it was one instruction repeated where it had no business being: a bounds check in front of every vector load.

This is a .NET performance post. The neural network is only the case study — you don't need to know anything about neural networks to follow it.

Measurement environment: Apple M3 Pro (11 cores), macOS 26.3, .NET 10.0.10, Arm64 RyuJIT armv8.0-a, BenchmarkDotNet 0.15.8, default job. Vector<float> is 4 wide here.

What Dense.Forward does: a SIMD dot product in C#

A dense layer computes, for each unit j, the dot product of its weights with the input, adds the bias, and applies the activation. Weights live in one flat array in per-unit order: unit j occupies Weights[j * Inputs .. (j+1) * Inputs].

The original version made a single pass, activating unit by unit:

for (int j = 0; j < Units; j++)
{
    float z = SimdOps.Dot(w.Slice(j * Inputs, Inputs), aIn) + Bias[j];
    aOut[j] = TActivation.Apply(z);
}

exp and tanh cost tens of cycles per call, and that loop makes one call per unit. The optimization looked obvious: split into two passes — first all the pre-activations, then the activation across the whole vector with TensorPrimitives — and process four values per instruction.

SimdOps.MatVec(Weights, aIn, Bias, aOut);   // dest[j] = dot(weights_j, x) + bias[j]
TActivation.ApplyAll(aOut);                 // activate the whole vector at once

In isolation, the optimization worked

Before touching the layer I measured the activations alone, in nanoseconds, scalar → vectorized:

At the widths that matter, roughly twice as fast. At width 10 the gain washes out and Softmax actually loses — 0.70× — because at that scale the fixed call cost dominates. An expected result, and in principle a good sign.

Integrated into the layer, it was a disaster

The 784×128 layer went from 12.79 µs to 80 µs. The arithmetic was identical; only the traversal order had changed.

Then came the data point no reasonable explanation covered: Dense<ReLU>, whose activation had not been touched, also measured 80 µs.

Worse, the same code called directly from a benchmark measured 10.7 µs. SimdOps.MatVec(_w, _in, _bias, _out) invoked from the benchmark took 10.51 µs. And _dense.Forward(_in, _out) — which does nothing but call MatVec and then the activation — took 79.78 µs.

To isolate that, I wrote a throwaway benchmark with seven variants of the same 784×128 layer:

This table comes from a temporary diagnostic harness, written to isolate the problem and deleted once it was solved. The numbers were genuinely measured, but that file is no longer in the repository: you cannot reproduce this table by cloning it. Every other measurement in this post you can.

The reading is unambiguous. The two-pass shape was not the problem: in isolation it was faster than the original — 11.49 µs against 12.91 µs. RealReference, a class with the same arithmetic, measured 12.78 µs. The disaster showed up only inside the real generic type.

Three hypotheses, three dead ends

The process matters as much as the conclusion, so here is what I tried and what failed.

Was TensorPrimitives.Tanh too large to inline? I marked Tanh.ApplyAll with [MethodImpl(MethodImplOptions.NoInlining)]. No change: 79.67 µs.

Field reloads from possible aliasing? I hoisted Inputs, Units, and Bias into locals so the JIT wouldn't have to re-read them. No change: 80.34 µs.

Compilation dropping to MinOpts? The JIT reported FullOpts, IL size 318, code size 1752. Fully optimized.

Three plausible explanations, three refuted by measurement. At that point there was nothing left to guess at from the source.

The disassembly: one bounds check per vector load

With DOTNET_JitDisasm dumping the real method, the inner loop was vectorized — the fmul and fadd over .4s were right there, exactly as expected. But every vector load was preceded by a bounds check:

cmp     x21, x11
bhi     G_M000_IG35          ; ← range check
lsl     xip0, xip0, #2
add     x22, x10, xip0
ldr     q18, [x22]
cmp     x21, w14, UXTW
bhi     G_M000_IG35          ; ← and another
add     xip0, x1, xip0
ldr     q19, [xip0]
fmul    v18.4s, v18.4s, v19.4s
fadd    v16.4s, v16.4s, v18.4s

Two compares and two conditional branches per pair of loads, inside the hottest loop in the program. That is what separates 10.7 µs from 79.78 µs.

The mechanism: inlined into Dense<TActivation>.Forward — a generic method that by then also contained the dot product and the activation, both inlined — the JIT stopped eliminating the bounds checks on the inner vector loads. In isolation, those same loads came out clean. As the host method grew, it lost the information that let the compiler prove the indices were safe.

Worth stating precisely: this is an observation reproducible on one version and one platform, not a declared .NET defect. I did not file it as a bug and I am not presenting it as one.

The fix, in one word: NoInlining

Pull the matrix-vector product into its own method — small, non-generic — and tell the JIT not to inline it:

[MethodImpl(MethodImplOptions.NoInlining)]
public static void MatVec(ReadOnlySpan<float> weights, ReadOnlySpan<float> x,
                          ReadOnlySpan<float> bias, Span<float> dest)
{
    int n = x.Length;

    for (int j = 0; j < dest.Length; j++)
        dest[j] = Dot(weights.Slice(j * n, n), x) + bias[j];
}

79.78 µs to 10.72 µs.

It is the opposite of the usual reflex. NoInlining reads like a pessimization; here it is what hands the JIT a method small enough to prove the accesses safe again.

The unexpected prize

The layer ended at 9.75 µs against the original 12.79 µs: 1.31× faster. That was the gain I had been after all along.

But the real finding is in the other row. Dense<ReLU> gained 1.29×, from 12.65 to 9.80 µs, with no change to its activation at all. Its original loop had been paying the same bounds-check penalty the whole time and nobody knew, because there had never been a measurement to compare it against. The 6× regression did not introduce the problem — it made it large enough to be impossible to ignore.

(The 12.65 µs figure is the pre-change measurement and now lives only in git history; the 9.80 µs one is published.)

The lesson for .NET performance work

A local optimization made the exact thing it optimized six times slower. Nothing in the source suggested it, every reasonable hypothesis I had turned out false, and the only thing that caught it was the benchmark.

Two rules I take from it:

1. If you inline vectorized code into a large generic method, measure. The code generator owes you nothing: guarantees it gives inside a small method do not necessarily survive inlining into a big one.
2. When the number doesn't add up, read the disassembly before inventing an explanation. I had three plausible explanations and all three were wrong. DOTNET_JitDisasm ruled them out in an afternoon.

And the caveat I can't leave out: all of these figures are ARM64 — M3 Pro, NEON, Vector<float> 4 wide — on .NET 10.0.10. On x86 with AVX2 or AVX-512, or on another JIT version, the effect may differ in magnitude or not appear at all.

The code

Everything above — except the diagnostic table, which no longer exists — is reproducible: neural-network-csharp. The benchmarks are in bench/, with the results discussed in bench/README.md.