Language Modeling on WikiText-103 (train)
15.18PPLBlock-ChaCAL
Evaluation Results
| Method | Links | |
|---|---|---|
| Block-ChaCALSetting=From scratch, block size=m=n//42026.05 | 15.18 | |
| Block-ChaCALSetting=From scratch, block size=m=n//32026.05 | 15.64 | |
| Block-ChaCALSetting=From scratch, block size=m=n//22026.05 | 15.96 | |
| GPT-2 (dense attention)Setting=From scratch2026.05 | 16.46 | |
| GPT-2 + ChaCALSetting=From scratch2026.05 | 17.57 | |
| LiMuonMemory (GB)=22.80, rank=full rank, Model=Mamba-130M2025.09 | 47.08 | |
| LiMuonMemory (GB)=22.31, rank=16, Model=Mamba-130M2025.09 | 50.26 | |
| Muon++Memory (GB)=22.35, Model=Mamba-130M2025.09 | 51.97 | |
| LiMuonMemory (GB)=20.25, rank=8, Model=Mamba-130M2025.09 | 59.07 | |
| MuonMemory (GB)=22.20, Model=Mamba-130M2025.09 | 62.49 | |
| MVN-Gradmodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.02 | 70.01 | |
| AdaBeliefmodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.02 | 70.92 | |
| Adammodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.02 | 71.33 | |
| LiMuonMemory (GB)=18.86, rank=4, Model=Mamba-130M2025.09 | 71.46 | |
| LaPropmodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.02 | 73.63 | |
| LionMemory (GB)=22.56, Model=Mamba-130M2025.09 | 75.25 | |
| SUMOMemory (GB)=20.10, Model=Mamba-130M2025.09 | 111.56 | |
| AdamWMemory (GB)=22.92, Model=Mamba-130M2025.09 | 253.75 | |
| AdamMemory (GB)=22.92, Model=Mamba-130M2025.09 | 480.33 |