# Parameter Reduction Improves Vision Transformers: A Comparative Study of Sharing and Width Reduction

Source: [Parameter Reduction Improves Vision Transformers: A Comparative Study of Sharing and Width Reduction](<https://www.sota2.com/research/paper/692e7305d6b0dd3342932689>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-03T05:04:47.680Z

Authors: Anantha Padmanaban Krishna Kumar

Published: 2025-11-30T20:04:30.000Z

## Abstract

Although scaling laws and many empirical results suggest that increasing the size of Vision Transformers often improves performance, model accuracy and training behavior are not always monotonically increasing with scale. Focusing on ViT-B/16 trained on ImageNet-1K, we study two simple parameter-reduction strategies applied to the MLP blocks, each removing 32.7\\% of the baseline parameters. Our \\emph{GroupedMLP} variant shares MLP weights between adjacent transformer blocks and achieves 81.47\\% top-1 accuracy while maintaining the baseline computational cost. Our \\emph{ShallowMLP} variant halves the MLP hidden dimension and reaches 81.25\\% top-1 accuracy with a 38\\% increase in inference throughput. Both models outperform the 86.6M-parameter baseline (81.05\\%) and exhibit substantially improved training stability, reducing peak-to-final accuracy degradation from 0.47\\% to the range 0.03\\% to 0.06\\%. These results suggest that, for ViT-B/16 on ImageNet-1K with a standard training recipe, the model operates in an overparameterized regime in which MLP capacity can be reduced without harming performance and can even slightly improve it. More broadly, our findings suggest that architectural constraints such as parameter sharing and reduced width may act as useful inductive biases, and highlight the importance of how parameters are allocated when designing Vision Transformers. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/parameter-efficient-vit-mlps.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2512.01059>)

[PDF](<https://arxiv.org/pdf/2512.01059.pdf>)