# Revisiting Structured Dropout

Source: [Revisiting Structured Dropout](<https://www.sota2.com/research/paper/68fa6d8c1e209131cea3a891>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:55.410Z

Authors: Yiren Zhao, Oluwatomisin Dada, Xitong Gao, Robert D Mullins

Published: 2022-10-05T21:26:57.000Z

## Abstract

Large neural networks are often overparameterised and prone to overfitting, Dropout is a widely used regularization technique to combat overfitting and improve model generalization. However, unstructured Dropout is not always effective for specific network architectures and this has led to the formation of multiple structured Dropout approaches to improve model performance and, sometimes, reduce the computational resources required for inference. In this work, we revisit structured Dropout comparing different Dropout approaches to natural language processing and computer vision tasks for multiple state-of-the-art networks. Additionally, we devise an approach to structured Dropout we call \\textbf{\\emph{ProbDropBlock}} which drops contiguous blocks from feature maps with a probability given by the normalized feature salience values. We find that with a simple scheduling strategy the proposed approach to structured Dropout consistently improved model performance compared to baselines and other Dropout approaches on a diverse range of tasks and models. In particular, we show \\textbf{\\emph{ProbDropBlock}} improves RoBERTa finetuning on MNLI by $0.22\\%$, and training of ResNet50 on ImageNet by $0.28\\%$.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2210.02570>)

[PDF](<https://arxiv.org/pdf/2210.02570.pdf>)
