Following the error backwards

Moving from one hidden neuron to two sounded like a tiny change. While learning to implement backpropagation, though, it made me realise I couldn’t treat the network as a single chain of calculations any more.

For a sigmoid output trained with binary cross-entropy, the gradient with respect to the output logit simplifies to p - y. From there, each hidden neuron gets its own learning signal, scaled by the weight connecting it to the output. The ReLU derivative can then block that signal before it reaches the neuron’s incoming weight and bias.

I’ve been writing these gradients out explicitly in Python rather than letting a framework calculate them. Keeping track of the separate paths takes more care than I expected, but I can now follow an error backwards through the network and see where each weight update comes from. That’s a much better foundation for building larger models than treating backpropagation as a black box.