Language Modeling on FineWeb-EDU (LM Loss)
2.89LM LossVeLO
Evaluation Results
| Method | Links | |
|---|---|---|
| VeLONumber of Parameters=355M2025.06 | 2.89 | |
| Adam + CosineNumber of Parameters=355M2025.06 | 2.91 | |
| μLOMNumber of Parameters=355M2025.06 | 3.18 | |
| Transformer (Dense baseline)Rank r=-, Backend N=47.25M, Interface N=-, Total N=63.63M2026.05 | 3.992 | |
| Hybrid (Dense baseline)Rank r=-, Backend N=60.97M, Interface N=-, Total N=85.55M2026.05 | 4.058 | |
| Mamba-2 (Dense baseline)Rank r=-, Backend N=51.33M, Interface N=-, Total N=75.91M2026.05 | 4.061 | |
| Mamba-3 SISO (Dense baseline)Rank r=-, Backend N=53.52M, Interface N=-, Total N=78.09M2026.05 | 4.07 | |
| Hybrid (LBI)Rank r=64, Backend N=60.97M, Interface N=8.32M, Total N=93.87M2026.05 | 4.221 | |
| Hybrid (LBI)Rank r=16, Backend N=60.97M, Interface N=7.84M, Total N=93.39M2026.05 | 4.223 | |
| Hybrid (LBI)Rank r=32, Backend N=60.97M, Interface N=8.00M, Total N=93.55M2026.05 | 4.237 | |
| Mamba-3 SISO (LBI)Rank r=32, Backend N=53.52M, Interface N=9.23M, Total N=87.33M2026.05 | 4.31 | |
| Mamba-3 SISO (LBI)Rank r=64, Backend N=53.52M, Interface N=9.60M, Total N=87.70M2026.05 | 4.312 | |
| Transformer (LBI)Rank r=32, Backend N=47.25M, Interface N=3.63M, Total N=67.26M2026.05 | 4.314 | |
| Transformer (LBI)Rank r=16, Backend N=47.25M, Interface N=3.52M, Total N=67.16M2026.05 | 4.318 | |
| Mamba-3 SISO (LBI)Rank r=16, Backend N=53.52M, Interface N=9.05M, Total N=87.14M2026.05 | 4.393 | |
| Mamba-2 (LBI)Rank r=32, Backend N=51.33M, Interface N=9.23M, Total N=85.14M2026.05 | 4.404 | |
| Mamba-2 (LBI)Rank r=64, Backend N=51.33M, Interface N=9.60M, Total N=85.51M2026.05 | 4.408 | |
| Mamba-2 (LBI)Rank r=16, Backend N=51.33M, Interface N=9.05M, Total N=84.95M2026.05 | 4.413 | |
| Transformer (LBI)Rank r=64, Backend N=47.25M, Interface N=3.84M, Total N=67.48M2026.05 | 4.646 |