How a Network Learns from a Mistake

If a network gives the wrong answer, what tells it which connection to change? The mistake appears at the end, but there might be thousands of connections upstream. I keep picturing each one raising its hand and claiming it helped.

That image catches the problem. The network doesn't identify a guilty connection. It calculates a gradient: for each weight, how would a tiny change affect the error? Backpropagation is the efficient way it works those effects backward from the output.

So it's not asking which connection caused the mistake in some human sense. It's asking how sensitive the mistake is to each connection right now. But if the signal starts at the end, how does it know what happened earlier?

It remembers values from the forward run: what entered each layer and what came out. Then it uses the chain rule. If one weight changes an intermediate value, and that value changes the loss, you multiply those two sensitivities.

Multiply them? I can follow that in words, but the backward direction still feels slippery. Could we try one tiny network, with no mystery layers? Maybe then I can see what the message going backward actually carries.

Sure. Let the input be one. One weight, set to two, multiplies it; a second weight, set to three, multiplies that result. The network says six, but we want four. For now, forget activation functions; this is just the bare wiring.

Okay, the middle value is two, and the final value is six. The output is two too high. Does backprop just send that two back to both weights, as if they share the blame equally?

Not quite. Choose a loss of half the squared gap from four. At six, its slope with respect to the output is two. Changing the second weight by a tiny amount changes the output about twice as much, so its loss gradient is four.

Right, because that second weight is multiplying the middle value, which is two. So the four is the sensitivity for that weight, not an amount to subtract automatically. What happens when the signal goes back through the second weight?

Changing the first weight slightly changes the middle value by that same amount, since the input is one. The second weight then triples that change. Multiply that three by the output loss slope, two, and the first weight's gradient is six.

Oh, so the earlier weight gets a larger gradient here, even though it's farther from the mistake. I was treating distance like blame. If both gradients are positive, do we simply turn both weights down?

Yes, with a small step. The gradient tells which direction would raise the loss locally, so an optimizer steps the other way. It might subtract a small multiple of six from the first weight and of four from the second, then run the network again.

Why run it again? If we've already worked out the gradients, couldn't we just keep stepping down? Or do the numbers six and four stop describing the network once the weights have moved?

Exactly. They describe the slopes at those particular weights. After the change, the middle value and the output shift, so the slopes can shift too. A big step can overshoot; repeated forward runs and backward passes let the network keep checking where it stands.

I see why the step has to be refreshed. But with thousands of weights, do we have to nudge each one separately just to measure its effect? That sounds like a huge number of extra runs.

No. That's the clever part of going backward. Each layer passes a loss sensitivity to the layer before it, using the values saved on the way forward. Shared intermediate calculations give gradients for all those weights without separately testing each one.

Our little network only multiplies. If real networks put other operations between weights, does that backward message still work? And could it get so faint across many layers that the early weights barely hear it?

Yes to both. A nonlinear operation has its own local slope, which backprop uses the same way. Across many layers, multiplying small slopes can make early gradients tiny, though network designs can help. So backprop traces sensitivity; the optimizer decides the step.