Residual Network

A Residual Network, commonly abbreviated as ResNet, is a deep convolutional neural network architecture introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun of Microsoft Research in 2015. ResNet addressed one of the central obstacles to training very deep neural networks — the degradation problem — through a simple but highly effective architectural innovation: skip connections, also called shortcut connections or residual connections.

By enabling gradients to flow directly through shortcut paths during backpropagation, ResNet made it practical to train networks with hundreds or even thousands of layers, far exceeding what had been possible with sequential architectures. ResNet won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2015 with a top-5 classification error of 3.57%, surpassing human-level performance on the benchmark for the first time.

The Degradation Problem

Prior to ResNet, empirical evidence had shown that adding more layers to a neural network did not reliably improve performance and often made it worse — not because of overfitting, but due to the degradation problem in training. As networks grew deeper, training accuracy would saturate and then decrease, even when the network theoretically had sufficient capacity to represent any shallower solution.

The root cause lay in backpropagation: as gradients were multiplied across many sequential layers, they became exponentially small (the vanishing gradient problem), making earlier layers extremely slow to learn. Batch normalisation had partially addressed this, but practical networks were still limited to tens of layers.

Residual Connections

ResNet's solution was the residual block: a building block in which the output of a block equals the learned transformation F(x) added to the block's input x directly:

This additive shortcut, where x is passed unchanged and added to the block's transformed output, is an identity mapping. The network is therefore learning the residual F(x) — the difference between the desired output and the identity — rather than the full mapping from input to output.

This formulation has two key consequences. First, if the optimal mapping for a particular block is close to the identity (i.e., the block should change the input very little), it is much easier to learn a near-zero F(x) than to learn an exact identity mapping through many non-linear layers. Second, and more importantly for training, the identity shortcut provides a direct path for gradients to flow from later layers back to earlier layers, effectively bypassing the multiplicative vanishing gradient problem across blocks.

When the input and output of a block have different dimensions (due to stride-based downsampling or increased channel depth), a linear projection is applied to the shortcut connection to match dimensions.

Architecture Variants

ResNet is available in several standard configurations differentiated by the number of layers:

| Variant | Layers | Block Type | Parameters (approx.) | |---|---|---|---| | ResNet-18 | 18 | Basic block | 11M | | ResNet-34 | 34 | Basic block | 21M | | ResNet-50 | 50 | Bottleneck block | 25M | | ResNet-101 | 101 | Bottleneck block | 44M | | ResNet-152 | 152 | Bottleneck block | 60M |

For ResNet-50 and deeper, the basic two-layer residual block is replaced with a three-layer bottleneck block: a 1x1 convolution that reduces channel dimensions, a 3x3 convolution, and a 1x1 convolution that restores dimensions. This reduces computational cost while allowing the network to go deeper.

Influence on Subsequent Architectures

ResNet's skip connection concept proved to be one of the most generative ideas in deep learning history. Virtually every major deep learning architecture developed after 2015 incorporates some form of residual or skip connection:

• DenseNet (2016): Extends the skip connection concept by connecting each layer to every subsequent layer within a dense block, creating a concatenated feature reuse.
• U-Net and Feature Pyramid Networks: Use skip connections between encoder and decoder paths for semantic segmentation and object detection.
• Transformer architectures: Every transformer block includes a residual connection around the attention sublayer and the feed-forward sublayer, making transformers a descendant of the residual network philosophy even in NLP.
• EfficientNet, ConvNeXt, ResNeXt: Build on the residual block paradigm with architectural improvements to scaling, grouped convolutions, and modern training techniques.

Training and Transfer Learning

Pre-trained ResNet models — particularly ResNet-50 and ResNet-101 — became foundational tools for transfer learning in computer vision. A ResNet pre-trained on ImageNet provides rich visual feature representations that can be fine-tuned for downstream tasks including medical image analysis, satellite imagery classification, industrial defect detection, and facial recognition. The Hugging Face model hub hosts hundreds of task-specific fine-tunes of ResNet variants.

ResNet in Practice Today

While newer architectures such as Vision Transformers (ViT) have surpassed ResNets on several benchmarks when trained on very large datasets, ResNets remain widely used in production systems due to their efficiency, interpretability, and well-understood behaviour. For edge deployment, smaller ResNet variants are frequently chosen because their convolutional structure is highly optimised for GPU and NPU inference. TensorFlow Lite, CoreML, and ONNX all provide optimised ResNet implementations for on-device inference.

Residual networks are extensively deployed in Malaysian manufacturing and industrial contexts. The Penang electronics and semiconductor manufacturing cluster — home to major operations from Intel, Bosch, Osram, and numerous EMS companies — uses computer vision systems for automated optical inspection (AOI) of printed circuit boards and semiconductor packages. ResNet-based architectures are commonly used as the feature extractor backbone in these inspection pipelines, with fine-tuning applied to detect specific defect classes relevant to each manufacturing process.

ViTrox Corporation Berhad, a Penang-based company listed on Bursa Malaysia and specialising in machine vision inspection systems, incorporates deep convolutional architectures related to or descended from ResNet in its AOI products. ViTrox serves customers across Southeast Asia and globally, making it one of the most prominent Malaysian companies in applied computer vision.

In the healthcare sector, Malaysian researchers at hospital AI units and university medical faculties have applied pre-trained ResNet models to medical image classification tasks including chest X-ray analysis for pneumonia and tuberculosis detection, diabetic retinopathy grading from fundus images, and histopathological slide analysis. Hospital Kuala Lumpur and several university teaching hospitals have run pilot programmes using AI-assisted radiology tools that use ResNet-derived architectures.

Universiti Malaya, Universiti Teknologi Malaysia, and Universiti Kebangsaan Malaysia have published research applying ResNet to Malaysian-specific challenges including oil palm disease identification from drone or satellite imagery, durian grading systems for the agricultural export market, and batik pattern classification for cultural heritage applications.

The MDEC Digital Economy initiatives, including the Smart Manufacturing Acceleration Programme, support SME manufacturers in adopting AI-based quality control, where ResNet-based visual inspection is one of the most mature and commercially accessible technologies. Grants under the Malaysia Digital Acceleration Grant (MDAG) have been used to fund integration of such vision systems by small and medium manufacturers in the Klang Valley and Johor regions.

1. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778.
2. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Identity mappings in deep residual networks. European Conference on Computer Vision (ECCV). Springer, Cham.
3. Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. CVPR 2017.
4. Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of ICML 2019.
5. Analytics Vidhya. (2023). Deep Residual Learning for Image Recognition: ResNet Explained. Analytics Vidhya Blog.