Depth-Learning Rate scaling analysis on ImageNet
-1.178Depth Exponent (alpha)ViT
Evaluation results
| Method | Links | |
|---|---|---|
| ViTOptimizer=SGD without momentum, Normalization=Post-LN, Patch Size=16, Embedding Dimension=768, Attention Heads=122026.02 | -1.178 | |
| CNNOptimizer=SGD without momentum, Activation=ReLU, Normalization=None, Dropout=None2026.02 | -1.329 | |
| Theory2026.02 | -1.5 | |
| ResNetOptimizer=SGD without momentum, Activation=ReLU, Normalization=None, Dropout=None, Initialization=Scaled He fan-in2026.02 | -1.567 |