Language Modeling on Fineweb 100B
2.6Perplexity (PPL)DELTA-PD
Evaluation Results
| Method | Links | |
|---|---|---|
| DELTA-PDParameters=230M, Layers=(3,6,9,12), Throughput=253k, Zero-shot=true2026.05 | 2.6 | |
| DeltaNetParameters=230M, Throughput=168k, Zero-shot=true2026.05 | 2.61 | |
| DELTA-PDParameters=230M, Layers=(6–12), Zero-shot=true2026.05 | 2.62 | |
| DELTA-PDParameters=230M, Layers=(1–6), Zero-shot=true2026.05 | 2.65 | |
| DELTA-PDParameters=100M, Layers=(3,6,9,12), Throughput=368k, Zero-shot=true2026.05 | 2.81 | |
| DELTA-PDParameters=100M, Layers=(6–12), Zero-shot=true2026.05 | 2.82 | |
| DELTA-PDParameters=100M, Layers=(1–6), Zero-shot=true2026.05 | 2.84 | |
| FLASH PD-SSMParameters=230M, Layers=(1–12), Zero-shot=true2026.05 | 2.84 | |
| DeltaNetParameters=100M, Throughput=350k, Zero-shot=true2026.05 | 2.85 | |
| FLASH PD-SSMParameters=100M, Layers=(1–12), Zero-shot=true2026.05 | 3.08 | |
| PathB4-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=55.67, Memory (GiB)=66.832026.03 | 12.29 | |
| PathB8-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=55.20, Memory (GiB)=66.942026.03 | 12.53 | |
| Indep-MoERouting=Independent Routing, Throughput (k tok/s/GPU)=55.21, Memory (GiB)=66.942026.03 | 12.91 | |
| Recurrent-MoERouting=Recurrent Routing, Throughput (k tok/s/GPU)=54.65, Memory (GiB)=69.262026.03 | 12.92 | |
| PathMoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=55.71, Memory (GiB)=66.742026.03 | 12.92 | |
| LowRank-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=53.84, Memory (GiB)=66.962026.03 | 12.98 | |
| X-MoERouting=Independent Routing, Throughput (k tok/s/GPU)=56.37, Memory (GiB)=67.062026.03 | 13.21 | |
| MonoB8-MoERouting=Path-Constrained Routing, Throughput (k tok/s/GPU)=56.53, Memory (GiB)=62.862026.03 | 13.34 | |
| Rand-MoERouting=Independent Routing, Throughput (k tok/s/GPU)=75.17, Memory (GiB)=47.492026.03 | 16.14 |