Language Modeling on FineWeb-Edu (test)
16.46Perplexity (Test)Mamba-3-MIMO-180M
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Mamba-3-MIMO-180MParams=180M, Tokens=100B, Zero-shot=true2026.05 | 16.46 | — | — | — | — | — | — | |
| Echo-180MParams=180M, Tokens=10B, Zero-shot=true2026.05 | 16.48 | — | — | — | — | — | — | |
| GDN-180MParams=180M, Tokens=100B, Zero-shot=true2026.05 | 16.52 | — | — | — | — | — | — | |
| Mamba-3-SISO-180MParams=180M, Tokens=100B, Zero-shot=true2026.05 | 16.59 | — | — | — | — | — | — | |
| Mamba-2-180MParams=180M, Tokens=100B, Zero-shot=true2026.05 | 16.76 | — | — | — | — | — | — | |
| Transformer-180MParams=180M, Tokens=100B, Zero-shot=true2026.05 | 16.89 | — | — | — | — | — | — | |
| BaseBudget=24x, Configuration=(24 ⊗ 1), Params=24x, FLOPs=24x2026.02 | 20.7 | — | — | — | — | — | — | |
| BaseBudget=12x, Configuration=(12 ⊗ 1), Params=12x, FLOPs=12x2026.02 | 22.24 | — | — | — | — | — | — | |
| TMLTBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 22.87 | — | — | — | — | — | — | |
| LoopFormerBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 22.87 | — | — | — | — | — | — | |
| TMLT-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 24.07 | — | — | — | — | — | — | |
| Base-LoopBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 24.53 | — | — | — | — | — | — | |
| DAG-MoE-lNumber of Parameters=699M2026.05 | 24.69 | — | — | — | — | — | — | |
| LoopFormerBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 25.02 | — | — | — | — | — | — | |
| BaseBudget=6x, Configuration=(6 ⊗ 1), Params=6x, FLOPs=6x2026.02 | 25.28 | — | — | — | — | — | — | |
| Base-Loop-EE-ConsBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 25.33 | — | — | — | — | — | — | |
| MoE-lNumber of Parameters=699M2026.05 | 25.38 | — | — | — | — | — | — | |
| LLaMAContext Length=512, h=512, L=12, Ex=339k, scale=Large-scale2026.02 | 25.8 | — | — | — | — | — | — | |
| MambaContext Length=512, h=512, L=12, Ex=339k, scale=Large-scale2026.02 | 26.5 | — | — | — | — | — | — | |
| Naive-Loop-EEBudget=24x, Configuration=(3 ⊗ 8), Params=3x, FLOPs=24x2026.02 | 26.55 | — | — | — | — | — | — | |
| Naive-Loop-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 26.74 | — | — | — | — | — | — | |
| Echo-50MParams=50M, Tokens=3B, Zero-shot=true2026.05 | 27.48 | — | — | — | — | — | — | |
| Base-Loop-EE-ConsBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 27.72 | — | — | — | — | — | — | |
| TMLT-EEBudget=12x, Configuration=(3 ⊗ 4), Params=3x, FLOPs=12x2026.02 | 28.28 | — | — | — | — | — | — | |
| Naive-Loop-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 29.36 | — | — | — | — | — | — | |
| ProtoTContext Length=512, h=512, L=12, Ex=339k, scale=Large-scale2026.02 | 29.5 | — | — | — | — | — | — | |
| DeltaNetContext Length=512, h=512, L=12, Ex=339k, scale=Large-scale2026.02 | 31.5 | — | — | — | — | — | — | |
| LoopFormerBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 33.45 | — | — | — | — | — | — | |
| Base-Loop-EE-ConsBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 35.95 | — | — | — | — | — | — | |
| TMLT-EEBudget=6x, Configuration=(3 ⊗ 2), Params=3x, FLOPs=6x2026.02 | 37.83 | — | — | — | — | — | — | |
| LLaMAContext Length=2048, h=256, L=6, Ex=18k2026.02 | 63.6 | — | — | — | — | — | — | |
| LLaMAContext Length=1024, h=256, L=6, Ex=18k2026.02 | 65.3 | — | — | — | — | — | — | |
| DeltaNetContext Length=2048, h=256, L=6, Ex=18k2026.02 | 68.9 | — | — | — | — | — | — | |
| MambaContext Length=2048, h=256, L=6, Ex=18k2026.02 | 69.5 | — | — | — | — | — | — | |
| LLaMAContext Length=512, h=256, L=6, Ex=18k2026.02 | 70.4 | — | — | — | — | — | — | |
| MambaContext Length=1024, h=256, L=6, Ex=18k2026.02 | 70.5 | — | — | — | — | — | — | |
| DeltaNetContext Length=1024, h=256, L=6, Ex=18k2026.02 | 70.6 | — | — | — | — | — | — | |
| ProtoTContext Length=2048, h=512, L=6, Ex=18k2026.02 | 73 | — | — | — | — | — | — | |
| ProtoTContext Length=1024, h=512, L=6, Ex=18k2026.02 | 73.7 | — | — | — | — | — | — | |
| ProtoTContext Length=1024, h=256, L=12, Ex=18k2026.02 | 74.6 | — | — | — | — | — | — | |
| DeltaNetContext Length=512, h=256, L=6, Ex=18k2026.02 | 76.3 | — | — | — | — | — | — | |
| ProtoTContext Length=2048, h=256, L=12, Ex=18k2026.02 | 76.7 | — | — | — | — | — | — | |
| MambaContext Length=512, h=256, L=6, Ex=18k2026.02 | 78 | — | — | — | — | — | — | |
| LLaMAContext Length=256, h=256, L=6, Ex=18k2026.02 | 78.7 | — | — | — | — | — | — | |
| ProtoTContext Length=1024, h=256, L=6, Ex=18k, R=642026.02 | 79.1 | — | — | — | — | — | — | |
| ProtoTContext Length=512, h=256, L=12, Ex=18k2026.02 | 79.8 | — | — | — | — | — | — | |
| ProtoTContext Length=1024, h=256, L=6, Ex=18k2026.02 | 80.5 | — | — | — | — | — | — | |
| ProtoTContext Length=2048, h=256, L=6, Ex=18k, R=642026.02 | 80.5 | — | — | — | — | — | — | |
| ProtoTContext Length=2048, h=256, L=6, Ex=18k2026.02 | 81.9 | — | — | — | — | — | — | |
| ProtoTContext Length=512, h=512, L=6, Ex=18k2026.02 | 82 | — | — | — | — | — | — | |
| ProtoTContext Length=512, h=256, L=6, Ex=18k, R=642026.02 | 83.2 | — | — | — | — | — | — | |
| ProtoTContext Length=512, h=256, L=6, Ex=18k2026.02 | 84.8 | — | — | — | — | — | — | |
| MambaContext Length=256, h=256, L=6, Ex=18k2026.02 | 86 | — | — | — | — | — | — | |
| DeltaNetContext Length=256, h=256, L=6, Ex=18k2026.02 | 90.4 | — | — | — | — | — | — | |
| ProtoTContext Length=256, h=256, L=6, Ex=18k2026.02 | 90.5 | — | — | — | — | — | — | |
| ProtoTContext Length=256, h=256, L=6, Ex=18k, R=642026.02 | 94 | — | — | — | — | — | — | |
| ProtoTContext Length=256, h=512, L=6, Ex=18k2026.02 | 97.2 | — | — | — | — | — | — | |
| ProtoTContext Length=256, h=256, L=12, Ex=18k2026.02 | 109.6 | — | — | — | — | — | — | |
| MelaParams=400M2026.05 | — | 12.53 | 12.75 | 12.01 | 12.64 | 14.43 | 14.5 | |
| MelaParams=800M2026.05 | — | 10.5 | 10.75 | 10.04 | 10.82 | 12.74 | 13.41 | |
| MelaParams=1.2B2026.05 | — | 9.08 | 9.49 | 9.2 | 9.91 | 11.64 | 11.85 | |
| Transformer++Params=400M2026.05 | — | 13.59 | 14.02 | 12.56 | 28.26 | 130.21 | 303.56 | |
| Transformer++Params=800M2026.05 | — | 12.17 | 12.42 | 11.35 | 16.69 | 129.1 | 497.85 | |
| Transformer++Params=1.2B2026.05 | — | 10.27 | 10.46 | 9.53 | 12.71 | 104.14 | 597.37 |