Language Modeling on WikiText-103 (Loss and Loss Gap)
3.156LossFull training
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Full trainingScale=900M2026.04 | 3.156 | — | |
| Full trainingScale=600M2026.04 | 3.185 | — | |
| Full trainingScale=300M2026.04 | 3.252 | — | |
| Full trainingScale=30M2026.04 | 3.757 | — | |
| LottaLoRAScale=900M, Rank=8, Trainable ratio=0.312026.04 | 3.95 | 0.79 | |
| LottaLoRAScale=600M, Rank=8, Trainable ratio=0.402026.04 | 4.058 | 0.87 | |
| LottaLoRAScale=300M, Rank=8, Trainable ratio=0.512026.04 | 4.191 | 0.94 | |
| LottaLoRAScale=30M, Rank=8, Trainable ratio=1.392026.04 | 4.681 | 0.92 | |
| Full trainingScale=3M2026.04 | 5.007 | — | |
| LottaLoRAScale=3M, Rank=8, Trainable ratio=7.682026.04 | 5.639 | 0.63 |