DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs
About
This paper revives Densely Connected Convolutional Networks (DenseNets) and reveals the underrated effectiveness over predominant ResNet-style architectures. We believe DenseNets' potential was overlooked due to untouched training methods and traditional design elements not fully revealing their capabilities. Our pilot study shows dense connections through concatenation are strong, demonstrating that DenseNets can be revitalized to compete with modern architectures. We methodically refine suboptimal components - architectural adjustments, block redesign, and improved training recipes towards widening DenseNets and boosting memory efficiency while keeping concatenation shortcuts. Our models, employing simple architectural elements, ultimately surpass Swin Transformer, ConvNeXt, and DeiT-III - key architectures in the residual learning lineage. Furthermore, our models exhibit near state-of-the-art performance on ImageNet-1K, competing with the very recent models and downstream tasks, ADE20k semantic segmentation, and COCO object detection/instance segmentation. Finally, we provide empirical analyses that uncover the merits of the concatenation over additive shortcuts, steering a renewed preference towards DenseNet-style designs. Our code is available at https://github.com/naver-ai/rdnet.
Key insights
What problem does RDNet address?
The paper revives DenseNets, which connect layers by concatenating features rather than adding them. It modernizes outdated training and design choices so concatenation-based networks compete with current residual architectures.
How does it differ from prior work?
RDNet updates DenseNet-style blocks, transition layers and growth rates. In a pilot study of 15,000 random architectures on Tiny-ImageNet, concatenation-based networks averaged 56.8% accuracy versus 55.9% for additive shortcuts; later experiments test the redesigned model at larger scales.
What practical uses do the findings support?
RDNet achieves competitive ImageNet-1K accuracy (82.8% for RDNet-T) and strong downstream results on ADE20K segmentation and COCO detection, plus robustness on out-of-distribution benchmarks. These are benchmark-level demonstrations, not documented production deployments.
What limitations do the authors note?
The authors state that models were scaled only to a large level, and resource constraints prevented extensions to upper scales such as ViT-G. They also note downstream hyperparameter searches were less extensive than ConvNeXt's, suggesting additional accuracy potential remains.