Language Modeling on FineWeb-Edu (val)
8.71PerplexityDeltaNet
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DeltaNetModel Scale=1.3B / 100B, Evaluation Mode=Zero-shot, Data Slice=matched packed train[-10000:]2026.05 | 8.71 | — | — | — | — | — | — | — | |
| OSDN-APFModel Scale=1.3B / 100B, Evaluation Mode=Zero-shot, Data Slice=matched packed train[-10000:]2026.05 | 9.1 | — | — | — | — | — | — | — | |
| MLRABranches=42026.03 | 9.193 | — | — | — | — | — | — | — | |
| MLRABranches=22026.03 | 9.242 | — | — | — | — | — | — | — | |
| GLABranches=22026.03 | 9.249 | — | — | — | — | — | — | — | |
| MLA2026.03 | 9.284 | — | — | — | — | — | — | — | |
| GLABranches=42026.03 | 9.307 | — | — | — | — | — | — | — | |
| TPA2026.03 | 9.333 | — | — | — | — | — | — | — | |
| GTA2026.03 | 9.346 | — | — | — | — | — | — | — | |
| GQA2026.03 | 9.427 | — | — | — | — | — | — | — | |
| MHA2026.03 | 9.434 | — | — | — | — | — | — | — | |
| MFA2026.03 | 9.506 | — | — | — | — | — | — | — | |
| MQA2026.03 | 9.533 | — | — | — | — | — | — | — | |
| MuownScale=2.7B, Learning Rate (η)=2 × 10−3, Weight Decay (λ)=02026.05 | 9.57 | — | — | — | — | — | — | — | |
| MuownScale=2.7B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=02026.05 | 9.63 | — | — | — | — | — | — | — | |
| MuownScale=2.7B, Learning Rate (η)=7.5 × 10−4, Weight Decay (λ)=02026.05 | 9.66 | — | — | — | — | — | — | — | |
| MuonScale=2.7B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=02026.05 | 9.82 | — | — | — | — | — | — | — | |
| MuonScale=2.7B, Learning Rate (η)=7.5 × 10−4, Weight Decay (λ)=02026.05 | 9.87 | — | — | — | — | — | — | — | |
| MuonScale=2.7B, Learning Rate (η)=7.5 × 10−4, Weight Decay (λ)=0.12026.05 | 9.96 | — | — | — | — | — | — | — | |
| MuonScale=2.7B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=0.12026.05 | 10.01 | — | — | — | — | — | — | — | |
| MuownScale=1B, Learning Rate (η)=3 × 10−3, Weight Decay (λ)=02026.05 | 11.9 | — | — | — | — | — | — | — | |
| MuownScale=1B, Learning Rate (η)=2 × 10−3, Weight Decay (λ)=02026.05 | 11.93 | — | — | — | — | — | — | — | |
| TransformerSize=1.3B2026.05 | 11.95 | — | — | — | — | — | — | — | |
| MuownScale=1B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=02026.05 | 12.01 | — | — | — | — | — | — | — | |
| Attractor ModelSize=770M2026.05 | 12.09 | — | — | — | — | — | — | — | |
| MuownScale=1B, Learning Rate (η)=5 × 10−4, Weight Decay (λ)=02026.05 | 12.18 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=2 × 10−3, Weight Decay (λ)=02026.05 | 12.2 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=0.12026.05 | 12.27 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=02026.05 | 12.32 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=2 × 10−3, Weight Decay (λ)=0.12026.05 | 12.33 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=5 × 10−4, Weight Decay (λ)=0.12026.05 | 12.35 | — | — | — | — | — | — | — | |
| SoftMuonModel=SmolLM2-360M, Batch size=512, Sequence length=1024, Epochs=12026.05 | 12.387 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=5 × 10−4, Weight Decay (λ)=02026.05 | 12.44 | — | — | — | — | — | — | — | |
| ParcaeSize=770M2026.05 | 12.49 | — | — | — | — | — | — | — | |
| MuonModel=SmolLM2-360M, Batch size=512, Sequence length=1024, Epochs=12026.05 | 12.541 | — | — | — | — | — | — | — | |
| AdamWScale=1B, Learning Rate (η)=2 × 10−3, Weight Decay (λ)=0.12026.05 | 12.69 | — | — | — | — | — | — | — | |
| AdamWScale=1B, Learning Rate (η)=5 × 10−4, Weight Decay (λ)=0.12026.05 | 12.85 | — | — | — | — | — | — | — | |
| AdamWScale=1B, Learning Rate (η)=1 × 10−3, Weight Decay (λ)=0.12026.05 | 12.93 | — | — | — | — | — | — | — | |
| TransformerSize=770M2026.05 | 13.08 | — | — | — | — | — | — | — | |
| Attractor ModelSize=370M2026.05 | 14.03 | — | — | — | — | — | — | — | |
| ParcaeSize=370M2026.05 | 14.49 | — | — | — | — | — | — | — | |
| TransformerSize=370M2026.05 | 15.79 | — | — | — | — | — | — | — | |
| Delta BlockScale=7.57B (d=4096, L=36), Steps=10K, Sequence Length=2048, Batch Size=64, Hardware=8×H100 FSDP, Precision=BF16, Software=torch.compile, Gradient Checkpointing=true2026.05 | 16 | 2.773 | — | — | — | 2.828 | 14,000 | 42.7 | |
| BaselineScale=7.57B (d=4096, L=36), Steps=10K, Sequence Length=2048, Batch Size=64, Hardware=8×H100 FSDP, Precision=BF16, Software=torch.compile, Gradient Checkpointing=true2026.05 | 17.43 | 2.858 | — | — | — | 2.918 | 21,400 | 41.6 | |
| Attractor ModelSize=140M2026.05 | 18.3 | — | — | — | — | — | — | — | |
| AttnResScale=7.57B (d=4096, L=36), Steps=10K, Sequence Length=2048, Batch Size=64, Hardware=8×H100 FSDP, Precision=BF16, Software=torch.compile, Gradient Checkpointing=true2026.05 | 18.58 | 2.922 | — | — | — | 2.984 | 12,500 | 44 | |
| ParcaeSize=140M2026.05 | 19.06 | — | — | — | — | — | — | — | |
| TransformerSize=140M2026.05 | 21.48 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=3 × 10−3, Weight Decay (λ)=0.12026.05 | 44.89 | — | — | — | — | — | — | — | |
| MuonScale=1B, Learning Rate (η)=3 × 10−3, Weight Decay (λ)=02026.05 | 110.19 | — | — | — | — | — | — | — | |
| AdamWScale=1B, Learning Rate (η)=3 × 10−3, Weight Decay (λ)=0.12026.05 | 303.16 | — | — | — | — | — | — | — | |
| AdamWScale=125M (5.24B tok), C (PFD)=0.0462026.05 | — | 3.3721 | — | — | — | — | — | — | |
| AdamWScale=399M (8.92B tok), C (PFD)=0.2472026.05 | — | 2.9966 | — | — | — | — | — | — | |
| AdamWScale=545M (14.04B tok), C (PFD)=0.5312026.05 | — | 2.9235 | — | — | — | — | — | — | |
| AdamWScale=1.1B (28.54B tok), C (PFD)=2.182026.05 | — | 2.7304 | — | — | — | — | — | — | |
| Baseline (Dense)Params (M)=67.11, Size (MB)=256.022026.02 | — | 4.3206 | 4.3206 | — | — | — | — | — | |
| BASISRank (R)=64, Seq. Memory Compression=1x (Sanity Check)2026.03 | — | 6.573 | — | — | — | — | — | — | |
| BASISRank (R)=32, Seq. Memory Compression=2x2026.03 | — | 6.575 | — | — | — | — | — | — | |
| BASISRank (R)=16, Seq. Memory Compression=4x2026.03 | — | 6.675 | — | — | — | — | — | — | |
| BASISRank (R)=8, Seq. Memory Compression=8x2026.03 | — | 6.762 | — | — | — | — | — | — | |
| BASISRank (R)=1, Seq. Memory Compression=64x (Maximal)2026.03 | — | 7.221 | — | — | — | — | — | — | |
| Exact BackpropagationRank (R)=N/A, Seq. Memory Compression=1x (None)2026.03 | — | 6.616 | — | — | — | — | — | — | |
| Hash-300KConfig=Hash-300K, Throughput (tok/s)=~1917, Params=313,567,2322026.01 | — | 4.4825 | — | — | 0.08 | — | — | — | |
| Hash-500KConfig=Hash-500K, Throughput (tok/s)=~1910, Params=313,567,2322026.01 | — | 4.4809 | — | 0.0082 | 0.07 | — | — | — | |
| Hash-800KConfig=Hash-800K, Throughput (tok/s)=~1917, Params=313,567,2322026.01 | — | 4.4961 | — | — | 0.08 | — | — | — | |
| HCModel Scale=M2026.03 | — | 3.288 | — | — | — | — | — | — | |
| HCModel Scale=L2026.03 | — | 3.111 | — | — | — | — | — | — | |
| mHCModel Scale=M2026.03 | — | 3.255 | — | — | — | — | — | — | |
| mHCModel Scale=L2026.03 | — | 3.009 | — | — | — | — | — | — | |
| mHC-liteModel Scale=M2026.03 | — | 3.254 | — | — | — | — | — | — | |
| mHC-liteModel Scale=L2026.03 | — | 3.006 | — | — | — | — | — | — | |
| Muon + MoonlightScale=125M (5.24B tok), C (PFD)=0.0462026.05 | — | 3.2319 | — | — | — | — | — | — | |
| Muon + MoonlightScale=399M (8.92B tok), C (PFD)=0.2472026.05 | — | 2.9183 | — | — | — | — | — | — | |
| Muon + MoonlightScale=545M (14.04B tok), C (PFD)=0.5312026.05 | — | 2.813 | — | — | — | — | — | — | |
| Muon + MoonlightScale=1.1B (28.54B tok), C (PFD)=2.182026.05 | — | 2.636 | — | — | — | — | — | — | |
| Nine-100/400KConfig=Nine-100/400K, Throughput (tok/s)=~1693, Params=313,567,2322026.01 | — | 4.4799 | — | 0.0123 | 0.1 | — | — | — | |
| Nine-20/480KConfig=Nine-20/480K, Throughput (tok/s)=~1670, Params=313,567,2322026.01 | — | 4.4872 | — | — | 0.17 | — | — | — | |
| Nine-50/450KConfig=Nine-50/450K, Throughput (tok/s)=~1670, Params=313,567,2322026.01 | — | 4.4942 | — | — | 0.12 | — | — | — | |
| OrScale-LMScale=125M (5.24B tok), C (PFD)=0.0462026.05 | — | 3.212 | — | — | — | — | — | — | |
| OrScale-LMScale=399M (8.92B tok), C (PFD)=0.2472026.05 | — | 2.9247 | — | — | — | — | — | — | |
| OrScale-LMScale=545M (14.04B tok), C (PFD)=0.5312026.05 | — | 2.8049 | — | — | — | — | — | — | |
| OrScale-LMScale=1.1B (28.54B tok), C (PFD)=2.182026.05 | — | 2.6251 | — | — | — | — | — | — | |
| RCModel Scale=M2026.03 | — | 3.325 | — | — | — | — | — | — | |
| RCModel Scale=L2026.03 | — | 3.048 | — | — | — | — | — | — | |
| sHCModel Scale=M2026.03 | — | 3.233 | — | — | — | — | — | — | |
| sHCModel Scale=L2026.03 | — | 3.003 | — | — | — | — | — | — | |
| Surgical HybridParams (M)=62.53, Size (MB)=238.542026.02 | — | 4.2838 | 4.2838 | — | — | — | — | — |