Language Modeling on FineWeb-EDU (train)
2.993LosssHC
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| sHCModel Scale=L2026.03 | 2.993 | — | — | — | |
| mHC-liteModel Scale=L2026.03 | 3.013 | — | — | — | |
| mHCModel Scale=L2026.03 | 3.017 | — | — | — | |
| RCModel Scale=L2026.03 | 3.058 | — | — | — | |
| HCModel Scale=L2026.03 | 3.112 | — | — | — | |
| sHCModel Scale=M2026.03 | 3.23 | — | — | — | |
| mHCModel Scale=M2026.03 | 3.241 | — | — | — | |
| mHC-liteModel Scale=M2026.03 | 3.241 | — | — | — | |
| HCModel Scale=M2026.03 | 3.276 | — | — | — | |
| RCModel Scale=M2026.03 | 3.313 | — | — | — | |
| BASISRank (R)=64, Seq. Memory Compression=1x (Sanity Check)2026.03 | 6.68 | — | — | — | |
| Exact BackpropagationRank (R)=N/A, Seq. Memory Compression=1x (None)2026.03 | 6.718 | — | — | — | |
| BASISRank (R)=16, Seq. Memory Compression=4x2026.03 | 6.731 | — | — | — | |
| BASISRank (R)=32, Seq. Memory Compression=2x2026.03 | 6.735 | — | — | — | |
| BASISRank (R)=8, Seq. Memory Compression=8x2026.03 | 6.798 | — | — | — | |
| BASISRank (R)=1, Seq. Memory Compression=64x (Maximal)2026.03 | 7.247 | — | — | — | |
| AdamWModel architecture=Llama3-1.8B2026.03 | — | 10.83 | 25.12 | 37.62 | |
| MuonModel architecture=Llama3-1.8B2026.03 | — | 10.01 | 25.39 | 32.51 | |
| NuMuonModel architecture=Llama3-1.8B2026.03 | — | 10.59 | 30.03 | 33.88 |