Language Modeling on WikiText-2 raw-v1 (test)
2.5Delta PPL (n=1)Interchange (clustered)
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Interchange (clustered)Backbone=Qwen3-8B, Precision=bf16, Words=5K, Window=512, Stride=2562026.05 | 2.5 | 10.3 | 45.4 | 22 | 21 | 39 | |
| SLEB-iterativeBackbone=Qwen3-8B, Precision=bf16, Words=5K, Window=512, Stride=2562026.05 | 2.5 | 11.2 | 46.2 | — | — | — | |
| BI-guidedBackbone=Qwen3-8B, Precision=bf16, Words=5K, Window=512, Stride=2562026.05 | 2.5 | 21.6 | 137.7 | — | — | — | |
| InterchangeBackbone=Pythia-1.4B, Precision=fp32, Words=5K, Window=512, Stride=2562026.05 | 11 | 61.8 | 377.8 | 66 | 36 | 16 | |
| BI-guidedBackbone=Pythia-1.4B, Precision=fp32, Words=5K, Window=512, Stride=2562026.05 | 11 | 74.7 | 186.9 | — | — | — | |
| ReplacementBackbone=Qwen3-8B, Precision=bf16, Words=5K, Window=512, Stride=2562026.05 | 11.3 | 48.2 | 116.5 | — | — | — | |
| SLEB-iterativeBackbone=Pythia-1.4B, Precision=fp32, Words=5K, Window=512, Stride=2562026.05 | 11.6 | 61.9 | 658.7 | — | — | — | |
| ReplacementBackbone=Pythia-1.4B, Precision=fp32, Words=5K, Window=512, Stride=2562026.05 | 16.7 | 172 | 2,310.6 | — | — | — |