I wrote the neural network study guide I wish I had found

When I learned how a neural network actually works inside, the material I found split into two piles. One explained the intuition without ever reaching the equations. The other gave the equations as if they were self-evident. Almost nothing explained why each piece exists: why there is an activation function, why the chain rule shows up where it does, why the order weights are stored in memory changes the runtime.

So I wrote that guide while building the implementation. It's 2,052 lines, it lives next to the code it describes, and every performance claim links to the measurement backing it.

How the guide is organized

Three parts, and the order matters.

Part I — the concepts, no code. From what problem a neural network actually solves through to the chain rule, by way of what a neuron is, what a hidden layer adds, and why learning is gradient descent. The chain rule appears when it's needed and applied to the specific case being built, not as a mathematical preamble.

Part II — the C# implementation. Every idea from Part I carried into real code: the data layout — "the most consequential decision" — the activations, the forward pass, SIMD, the backward pass, weight initialization, the training loop, and the gradient check.

Part III — practice. Results, a debugging manual, sixteen exercises, what the implementation does not do, and where to go next.

There's also an explicit recommendation up front for anyone in a hurry: if you only read three sections, read §7, §8, and §21.

Backpropagation worked by hand, with a calculator

§7 and §8 are complete worked examples: one neuron by hand, then the full chain through two layers. Small numbers, every step written out, calculated and verified.

That format matters more than it sounds. The difference between understanding backpropagation and believing you understand it usually comes down to whether you can compute a two-layer gradient by hand. If the numbers come out, the abstraction holds afterwards. If they don't, you carry an approximate intuition around for months that breaks the moment something goes wrong.

§21 — gradient checking, the section almost nobody writes

A subtly wrong backward pass usually still trains. Loss goes down, nothing blows up, and the results are mediocre in a way that's indistinguishable from "the hyperparameters need tuning." Those bugs cost days.

The defense is comparing the analytic gradient against a numerical estimate that doesn't use your code: nudge a weight by ±ε, measure how the loss actually moves, compare. Slow — two full forward passes per parameter — but decisive.

What makes that section useful isn't the formula, it's the table:

The U shape is the evidence. Two error sources fight each other: a large ε approximates a derivative badly, and a small ε subtracts two nearly equal floats and loses precision. A correct gradient shows that tradeoff with an optimum in the middle. An incorrect gradient shows O(1) error at any ε, because it isn't approximating anything. You don't have to trust a threshold — you look at the shape.

And it ships with its documented exception, which is the part that would have saved me the most at the time: ReLU can fail the check while the code is correct. Finite differences assume the loss is smooth between w−ε and w+ε, and ReLU has a kink at z = 0. If the step crosses that corner, the numerical estimate measures the average of two different slopes. Measured on exactly that setup — a unit with z 0.001 from the kink — the error is 2.9e-1 at ε = 1e-2, dropping to 8.8e-2 at ε = 1e-4. There, the analytic gradient is right and the estimate is what's wrong.

That yields two diagnostics for telling a kink from a real bug: shrink ε — an artifact improves sharply, a bug doesn't move — and rebuild the same network with tanh, which has no corner to cross. Both behaviors are pinned by tests.

Three things that set this from-scratch neural network in C# apart

1. It verifies its own backward pass, and proves the verification works. The suite includes a deliberately wrong derivative — a Tanh variant using 1 + a² instead of 1 - a² — that must produce O(1) error. Without it, passing tests could be passing vacuously: a check that cannot fail proves nothing.

2. It measures its own claims and publishes the one that turned out false. The repo claimed an activation invoked through a delegate would be clearly slower than a generic one. The measurement says no: it lands within ±4 % at every realistic size, and the sign isn't even consistent. The documentation corrects it rather than quietly deleting the sentence.

That was the cheapest claim to correct. I've written about the two that cost more: an optimization that made a layer 6.3× slower and that only the disassembly accounted for, and a benchmark that turned out to be noise while I had already written a convincing explanation of that noise. There's also the case where the standard library won one operation and lost the other.

3. It's honest about its limits. §25 lists eight things the implementation doesn't do, ordered by what they cost. The first is the most uncomfortable: ForwardBatch measures 0.98× against a manual loop — a deliberate null result, published as such — and a real blocked GEMM would likely dominate any other optimization in the current code. Which is to say, the entire axis I've been optimizing along is the secondary one, and that's said before anything else.

The break-it exercises

The sixteen exercises are ordered by value, and the ones that teach most are the ones that break something on purpose. Zeroing the weights and watching XOR freeze at a loss of exactly 0.250000, predicting 0.5000 forever. Changing Tanh's derivative and watching the check jump to O(1) while training still partly works. Re-enabling the forward-pass cache at the wrong moment and confirming that the loss keeps dropping while the network trains on the wrong example.

That last one is, I think, the central argument of the whole project: "it still trains" proves nothing.

The result: MNIST at 98.02 % in 37 seconds

Environment: Apple M3 Pro (11 cores), macOS 26.3, .NET 10.0.10, Arm64 RyuJIT armv8.0-a.

Thirty-seven seconds matters for a concrete reason: you can change something, retrain, and see the effect inside the same minute. That short loop is what makes a guide something you can use while reading it. And the baseline is published precisely so it can be beaten: exercise 10 asks you to implement momentum and measure it against that 98.02 % in 37 s.

Fully bilingual, including the benchmarks

The project is completely duplicated in English and Spanish: README.md / README.es.md, STUDY-GUIDE.md / STUDY-GUIDE.es.md, bench/README.md / bench/README.es.md. The guide runs 2,052 lines in English and 2,152 in Spanish — the Spanish is not a summary. Technical documentation at this depth about .NET and performance is scarce in Spanish, which is why the duplication was worth the effort.

What stays in English on purpose, in both versions, is the code, the identifiers, the command-line flags, and the program output: those are the repository's real artifacts, and translating them would describe a program that doesn't exist.

The caveat

This project is educational. It's single-threaded, has no real batching, no blocked GEMM, and uses plain SGD without momentum or Adam, and it does not aim to compete with any specialized tool. It's optimized only as far as optimizing taught something.

What it does offer is a property that doesn't always come with educational material: everything it claims is measured, and what turned out false is still published. That's the part I wish I had found.

The code

Everything above is reproducible: neural-network-csharp. The study guide is STUDY-GUIDE.md, and the benchmarks are in bench/, with the results discussed in bench/README.md.