Language Modeling on WikiText and LAMBADA (test)
7.3WikiText PPLLlama3 8B
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Llama3 8BCR=uncompressed, Backbone=Llama3-8B2026.02 | 7.3 | 3.1 | |
| ROCKETCR=0.2, Backbone=Llama3-8B2026.02 | 11 | 3.8 | |
| DNH-HOPEModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Static setting2025.11 | 14.92 | 10.87 | |
| ROCKETCR=0.3, Backbone=Llama3-8B2026.02 | 15 | 4.4 | |
| Titans (LMM)Model Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Static setting2025.11 | 15.6 | 11.41 | |
| DeltaNetModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Static setting2025.11 | 17.71 | 16.88 | |
| Transformer++Model Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Static setting2025.11 | 18.53 | 18.32 | |
| RetNetModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Static setting2025.11 | 19.08 | 17.27 | |
| DNH-HOPEModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Static setting2025.11 | 19.82 | 20.15 | |
| CoSpaDiCR=0.2, Backbone=Llama3-8B2026.02 | 20 | 4.3 | |
| Titans (LMM)Model Parameters=760M, Training Tokens=30B, Evaluation Protocol=Static setting2025.11 | 20.04 | 21.96 | |
| HOPEModel Parameters=1.3B, Training Tokens=100B, Evaluation Protocol=Static setting2025.11 | 20.53 | 20.47 | |
| DeltaNetModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Static setting2025.11 | 24.37 | 24.6 | |
| Transformer++Model Parameters=760M, Training Tokens=30B, Evaluation Protocol=Static setting2025.11 | 25.21 | 27.64 | |
| HOPEModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Static setting2025.11 | 26.05 | 29.38 | |
| RetNetModel Parameters=760M, Training Tokens=30B, Evaluation Protocol=Static setting2025.11 | 26.08 | 24.45 | |
| ROCKETCR=0.4, Backbone=Llama3-8B2026.02 | 29 | 8.1 | |
| SVD-LLMCR=0.2, Backbone=Llama3-8B2026.02 | 41 | 11 | |
| CoSpaDiCR=0.3, Backbone=Llama3-8B2026.02 | 45 | 9.2 | |
| SVD-LLMCR=0.3, Backbone=Llama3-8B2026.02 | 150 | 61 | |
| CoSpaDiCR=0.4, Backbone=Llama3-8B2026.02 | 180 | 120 | |
| SVD-LLMCR=0.4, Backbone=Llama3-8B2026.02 | 550 | 1,300 | |
| CoSpaDiCR=0.5, Backbone=Llama3-8B2026.02 | 740 | 520 | |
| SVD-LLMCR=0.5, Backbone=Llama3-8B2026.02 | 1,800 | 6,500 |