ResNets and the Depth Breakthrough

A 2015 paper from Microsoft Research let neural networks go 150 layers deep by adding a shortcut.

28 min · Reviewed 2026

The Depth Problem

After AlexNet in 2012, everyone wanted deeper networks. VGG went 16 to 19 layers in 2014. GoogLeNet went 22. But researchers kept hitting a wall: making networks deeper made them worse, not better, even on training data.

In December 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research Asia published Deep Residual Learning for Image Recognition. They trained networks 152 layers deep and won ImageNet by a comfortable margin.

Why this fixed depth

Gradients could flow directly through skip connections, dodging the vanishing gradient problem
Identity mappings became a safe default; layers could learn to do nothing if that was best
Optimization got much easier, so deeper became genuinely better
Performance scaled almost monotonically with depth, up to 1000 layers in follow-up work

ResNets were a contagious idea. Skip connections or variants of them now appear in almost every successful architecture: U-Nets for segmentation, DenseNets, and crucially Transformers, which use residual connections around each attention block.

Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the vanishing or exploding gradient problem.
— He et al., 2015

The big idea: a tiny architectural tweak unlocked an order of magnitude of depth. The lesson, repeated throughout AI, is that the right small idea compounds enormously.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-history-resnets-creators

What is the core idea behind "ResNets and the Depth Breakthrough"?
1. A 2015 paper from Microsoft Research let neural networks go 150 layers deep by adding a shortcut.
2. Fifth Generation
3. behaviorism
4. Other minds reply: we have no independent evidence that humans understand either…
Which term best describes a foundational idea in "ResNets and the Depth Breakthrough"?
1. residual connection
2. ResNet
3. skip connection
4. vanishing gradient
A learner studying ResNets and the Depth Breakthrough would need to understand which concept?
1. ResNet
2. skip connection
3. residual connection
4. vanishing gradient
Which of these is directly relevant to ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. vanishing gradient
4. skip connection
Which of the following is a key point about ResNets and the Depth Breakthrough?
1. Gradients could flow directly through skip connections, dodging the vanishing gradient problem
2. Identity mappings became a safe default; layers could learn to do nothing if that was best
3. Optimization got much easier, so deeper became genuinely better
4. Performance scaled almost monotonically with depth, up to 1000 layers in follow-up work
Which of these does NOT belong in a discussion of ResNets and the Depth Breakthrough?
1. Optimization got much easier, so deeper became genuinely better
2. Identity mappings became a safe default; layers could learn to do nothing if that was best
3. Gradients could flow directly through skip connections, dodging the vanishing gradient problem
4. Fifth Generation
What is the key insight about "The one-line idea" in the context of ResNets and the Depth Breakthrough?
1. Fifth Generation
2. behaviorism
3. Instead of asking each layer to learn the full transformation, ask it to learn the residual, the difference between the …
4. Other minds reply: we have no independent evidence that humans understand either…
Which statement accurately describes an aspect of ResNets and the Depth Breakthrough?
1. Fifth Generation
2. behaviorism
3. Other minds reply: we have no independent evidence that humans understand either…
4. After AlexNet in 2012, everyone wanted deeper networks. VGG went 16 to 19 layers in 2014. GoogLeNet went 22.
What does working with ResNets and the Depth Breakthrough typically involve?
1. In December 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research Asia published Deep Residual Learning for Imag…
2. Fifth Generation
3. behaviorism
4. Other minds reply: we have no independent evidence that humans understand either…
Which of the following is true about ResNets and the Depth Breakthrough?
1. Fifth Generation
2. ResNets were a contagious idea. Skip connections or variants of them now appear in almost every successful architecture: U-Nets for segmenta…
3. behaviorism
4. Other minds reply: we have no independent evidence that humans understand either…
Which best describes the scope of "ResNets and the Depth Breakthrough"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It focuses on A 2015 paper from Microsoft Research let neural networks go 150 layers deep by adding a shortcut.
4. It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about ResNets and the Depth Breakthrough?
1. Fifth Generation
2. behaviorism
3. Other minds reply: we have no independent evidence that humans understand either…
4. Why this fixed depth
Which of the following is a concept covered in ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. skip connection
4. vanishing gradient
Which of the following is a concept covered in ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. skip connection
4. vanishing gradient
Which of the following is a concept covered in ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. skip connection
4. vanishing gradient

← Back to interactive lesson

Tendril · Creators · AI Foundations

ResNets and the Depth Breakthrough

A 2015 paper from Microsoft Research let neural networks go 150 layers deep by adding a shortcut.

28 min · Reviewed 2026

The Depth Problem

Why this fixed depth

Gradients could flow directly through skip connections, dodging the vanishing gradient problem
Identity mappings became a safe default; layers could learn to do nothing if that was best
Optimization got much easier, so deeper became genuinely better
Performance scaled almost monotonically with depth, up to 1000 layers in follow-up work

Is learning better networks as easy as stacking more layers? An obstacle to answering this question was the vanishing or exploding gradient problem.
— He et al., 2015

The big idea: a tiny architectural tweak unlocked an order of magnitude of depth. The lesson, repeated throughout AI, is that the right small idea compounds enormously.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-history-resnets-creators

What is the core idea behind "ResNets and the Depth Breakthrough"?
1. A 2015 paper from Microsoft Research let neural networks go 150 layers deep by adding a shortcut.
2. Fifth Generation
3. behaviorism
4. Other minds reply: we have no independent evidence that humans understand either…
Which term best describes a foundational idea in "ResNets and the Depth Breakthrough"?
1. residual connection
2. ResNet
3. skip connection
4. vanishing gradient
A learner studying ResNets and the Depth Breakthrough would need to understand which concept?
1. ResNet
2. skip connection
3. residual connection
4. vanishing gradient
Which of these is directly relevant to ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. vanishing gradient
4. skip connection
Which of the following is a key point about ResNets and the Depth Breakthrough?
1. Gradients could flow directly through skip connections, dodging the vanishing gradient problem
2. Identity mappings became a safe default; layers could learn to do nothing if that was best
3. Optimization got much easier, so deeper became genuinely better
4. Performance scaled almost monotonically with depth, up to 1000 layers in follow-up work
Which of these does NOT belong in a discussion of ResNets and the Depth Breakthrough?
1. Optimization got much easier, so deeper became genuinely better
2. Identity mappings became a safe default; layers could learn to do nothing if that was best
3. Gradients could flow directly through skip connections, dodging the vanishing gradient problem
4. Fifth Generation
What is the key insight about "The one-line idea" in the context of ResNets and the Depth Breakthrough?
1. Fifth Generation
2. behaviorism
3. Instead of asking each layer to learn the full transformation, ask it to learn the residual, the difference between the …
4. Other minds reply: we have no independent evidence that humans understand either…
Which statement accurately describes an aspect of ResNets and the Depth Breakthrough?
1. Fifth Generation
2. behaviorism
3. Other minds reply: we have no independent evidence that humans understand either…
4. After AlexNet in 2012, everyone wanted deeper networks. VGG went 16 to 19 layers in 2014. GoogLeNet went 22.
What does working with ResNets and the Depth Breakthrough typically involve?
1. In December 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at Microsoft Research Asia published Deep Residual Learning for Imag…
2. Fifth Generation
3. behaviorism
4. Other minds reply: we have no independent evidence that humans understand either…
Which of the following is true about ResNets and the Depth Breakthrough?
1. Fifth Generation
2. ResNets were a contagious idea. Skip connections or variants of them now appear in almost every successful architecture: U-Nets for segmenta…
3. behaviorism
4. Other minds reply: we have no independent evidence that humans understand either…
Which best describes the scope of "ResNets and the Depth Breakthrough"?
1. It is unrelated to foundations workflows
2. It applies only to the opposite beginner tier
3. It focuses on A 2015 paper from Microsoft Research let neural networks go 150 layers deep by adding a shortcut.
4. It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about ResNets and the Depth Breakthrough?
1. Fifth Generation
2. behaviorism
3. Other minds reply: we have no independent evidence that humans understand either…
4. Why this fixed depth
Which of the following is a concept covered in ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. skip connection
4. vanishing gradient
Which of the following is a concept covered in ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. skip connection
4. vanishing gradient
Which of the following is a concept covered in ResNets and the Depth Breakthrough?
1. ResNet
2. residual connection
3. skip connection
4. vanishing gradient

← Back to interactive lesson