Backpropagation Rediscovered, 1986

In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published Learning representations by back-propagating errors in Nature. Their paper showed how to train multi-layer neural networks by using the chain rule to propagate error gradients from output back to input.

How backprop works, in one paragraph

1. 1Forward pass: compute the prediction layer by layer
2. 2Compute the loss against the true label
3. 3Backward pass: compute gradients from output back to input
4. 4Update weights by a small step against the gradient
5. 5Repeat across the dataset, many times

“It was a really good feeling when we realized this was going to work.”

What backprop needed to fully arrive

• Massive labeled datasets, eventually provided by the internet
• Parallel compute, eventually provided by NVIDIA GPUs
• Better activation functions like ReLU, which dodged vanishing gradients
• Patience from a handful of researchers, especially Hinton, Bengio, and LeCun

The big idea: the central algorithm of modern AI was published in 1986 and then sat mostly dormant for a generation. Sometimes the bottleneck is not the math but the hardware and data around it.