List your company
ResearchFormerly WizWand

Language Modeling on Fineweb 100B

2.6Perplexity (PPL)DELTA-PD
2.05845.71429.3713.0258Mar 18, 2026Mar 28, 2026Apr 7, 2026Apr 17, 2026Apr 27, 2026May 7, 2026May 18, 2026
Updated 5mo ago

Evaluation results

MethodLinks
DELTA-PDParameters=230M, Layers=(3,6,9,12), Throughput=253k, Zero-shot=true2026.052.6
DeltaNetParameters=230M, Throughput=168k, Zero-shot=true2026.052.61
DELTA-PDParameters=230M, Layers=(6–12), Zero-shot=true2026.052.62
DELTA-PDParameters=230M, Layers=(1–6), Zero-shot=true2026.052.65
DELTA-PDParameters=100M, Layers=(3,6,9,12), Throughput=368k, Zero-shot=true2026.052.81
DELTA-PDParameters=100M, Layers=(6–12), Zero-shot=true2026.052.82
DELTA-PDParameters=100M, Layers=(1–6), Zero-shot=true2026.052.84
FLASH PD-SSMParameters=230M, Layers=(1–12), Zero-shot=true2026.052.84
DeltaNetParameters=100M, Throughput=350k, Zero-shot=true2026.052.85
FLASH PD-SSMParameters=100M, Layers=(1–12), Zero-shot=true2026.053.08
PathB4-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=55.67, Memory (GiB)=66.832026.0312.29
PathB8-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=55.20, Memory (GiB)=66.942026.0312.53
Indep-MoERouting=Independent Routing, Throughput (k tok/s/GPU)=55.21, Memory (GiB)=66.942026.0312.91
Recurrent-MoERouting=Recurrent Routing, Throughput (k tok/s/GPU)=54.65, Memory (GiB)=69.262026.0312.92
PathMoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=55.71, Memory (GiB)=66.742026.0312.92
LowRank-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=53.84, Memory (GiB)=66.962026.0312.98
X-MoERouting=Independent Routing, Throughput (k tok/s/GPU)=56.37, Memory (GiB)=67.062026.0313.21
MonoB8-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=56.53, Memory (GiB)=62.862026.0313.34
Rand-MoERouting=Independent Routing, Throughput (k tok/s/GPU)=75.17, Memory (GiB)=47.492026.0316.14