List your company
ResearchFormerly WizWand

Language Modeling on FineWeb-EDU (LM Loss)

2.89LM LossVeLO
2.819763.293883.7684.24212Jun 12, 2025Aug 6, 2025Sep 30, 2025Nov 24, 2025Jan 18, 2026Mar 14, 2026May 9, 2026
Updated 5mo ago

Evaluation results

MethodLinks
VeLONumber of Parameters=355M2025.062.89
Adam + CosineNumber of Parameters=355M2025.062.91
μLOMNumber of Parameters=355M2025.063.18
Transformer (Dense baseline)Rank r=-, Backend N=47.25M, Interface N=-, Total N=63.63M2026.053.992
Hybrid (Dense baseline)Rank r=-, Backend N=60.97M, Interface N=-, Total N=85.55M2026.054.058
Mamba-2 (Dense baseline)Rank r=-, Backend N=51.33M, Interface N=-, Total N=75.91M2026.054.061
Mamba-3 SISO (Dense baseline)Rank r=-, Backend N=53.52M, Interface N=-, Total N=78.09M2026.054.07
Hybrid (LBI)Rank r=64, Backend N=60.97M, Interface N=8.32M, Total N=93.87M2026.054.221
Hybrid (LBI)Rank r=16, Backend N=60.97M, Interface N=7.84M, Total N=93.39M2026.054.223
Hybrid (LBI)Rank r=32, Backend N=60.97M, Interface N=8.00M, Total N=93.55M2026.054.237
Mamba-3 SISO (LBI)Rank r=32, Backend N=53.52M, Interface N=9.23M, Total N=87.33M2026.054.31
Mamba-3 SISO (LBI)Rank r=64, Backend N=53.52M, Interface N=9.60M, Total N=87.70M2026.054.312
Transformer (LBI)Rank r=32, Backend N=47.25M, Interface N=3.63M, Total N=67.26M2026.054.314
Transformer (LBI)Rank r=16, Backend N=47.25M, Interface N=3.52M, Total N=67.16M2026.054.318
Mamba-3 SISO (LBI)Rank r=16, Backend N=53.52M, Interface N=9.05M, Total N=87.14M2026.054.393
Mamba-2 (LBI)Rank r=32, Backend N=51.33M, Interface N=9.23M, Total N=85.14M2026.054.404
Mamba-2 (LBI)Rank r=64, Backend N=51.33M, Interface N=9.60M, Total N=85.51M2026.054.408
Mamba-2 (LBI)Rank r=16, Backend N=51.33M, Interface N=9.05M, Total N=84.95M2026.054.413
Transformer (LBI)Rank r=64, Backend N=47.25M, Interface N=3.84M, Total N=67.48M2026.054.646