# Big Neural Networks Waste Capacity

Source: [Big Neural Networks Waste Capacity](<https://www.sota2.com/research/paper/68fa60d81e209131ce9f1568>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T19:47:33.519Z

Authors: Yann N. Dauphin, Yoshua Bengio

Published: 2013-01-16T04:45:29.000Z

## Abstract

This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1301.3583>)

[PDF](<https://arxiv.org/pdf/1301.3583.pdf>)