ResNets and the Depth Breakthrough

After AlexNet in 2012, everyone wanted deeper networks. VGG went 16 to 19 layers in 2014. GoogLeNet went 22. But researchers kept hitting a wall: making networks deeper made them worse, not better, even on training data.

In December 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research Asia published Deep Residual Learning for Image Recognition. They trained networks 152 layers deep and won ImageNet by a comfortable margin.

Why this fixed depth

• Gradients could flow directly through skip connections, dodging the vanishing gradient problem
• Identity mappings became a safe default; layers could learn to do nothing if that was best
• Optimization got much easier, so deeper became genuinely better
• Performance scaled almost monotonically with depth, up to 1000 layers in follow-up work

ResNets were a contagious idea. Skip connections or variants of them now appear in almost every successful architecture: U-Nets for segmentation, DenseNets, and crucially Transformers, which use residual connections around each attention block.

“Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the vanishing or exploding gradient problem.”

The big idea: a tiny architectural tweak unlocked an order of magnitude of depth. The lesson, repeated throughout AI, is that the right small idea compounds enormously.