List your company
ResearchFormerly WizWand

Language Modeling on WikiText-103 (train)

15.18PPLBlock-ChaCAL
-3.426122.1645247.755373.3455Sep 18, 2025Oct 28, 2025Dec 8, 2025Jan 18, 2026Feb 28, 2026Apr 10, 2026May 21, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Block-ChaCALSetting=From scratch, block size=m=n//42026.0515.18
Block-ChaCALSetting=From scratch, block size=m=n//32026.0515.64
Block-ChaCALSetting=From scratch, block size=m=n//22026.0515.96
GPT-2 (dense attention)Setting=From scratch2026.0516.46
GPT-2 + ChaCALSetting=From scratch2026.0517.57
LiMuonMemory (GB)=22.80, rank=full rank, Model=Mamba-130M2025.0947.08
LiMuonMemory (GB)=22.31, rank=16, Model=Mamba-130M2025.0950.26
Muon++Memory (GB)=22.35, Model=Mamba-130M2025.0951.97
LiMuonMemory (GB)=20.25, rank=8, Model=Mamba-130M2025.0959.07
MuonMemory (GB)=22.20, Model=Mamba-130M2025.0962.49
MVN-Gradmodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.0270.01
AdaBeliefmodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.0270.92
Adammodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.0271.33
LiMuonMemory (GB)=18.86, rank=4, Model=Mamba-130M2025.0971.46
LaPropmodel=GPT-2, sequence length=512, batch size=128, regularization=coupled l2, seeds=42026.0273.63
LionMemory (GB)=22.56, Model=Mamba-130M2025.0975.25
SUMOMemory (GB)=20.10, Model=Mamba-130M2025.09111.56
AdamWMemory (GB)=22.92, Model=Mamba-130M2025.09253.75
AdamMemory (GB)=22.92, Model=Mamba-130M2025.09480.33