Long-context recall and reasoning on RULER 4K context lengths
100MK1 ScoreLlama-3.1-8B
Evaluation Results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8BArchitecture=Transformer, Samples per task=502025.05 | 100 | 100 | 98 | 90.5 | 98.5 | 99.6 | 95.2 | 91.7 | 62 | 81.5 | 91.7 | |
| Mamba2-8BArchitecture=Subquadratic2025.05 | 40.3 | 13.8 | 5.5 | 49.1 | 35 | 76.5 | 32.9 | 76.6 | 31.8 | 35.5 | 39.7 | |
| LoLA-8BArchitecture=Subquadratic, η=128, λ=7682025.05 | 39.4 | 11.6 | 7.6 | 67.6 | 65 | 85.2 | 45.9 | 51.3 | 24.2 | 53.9 | 45.2 | |
| LoLCATs-8B+Architecture=Subquadratic, η=8962025.05 | 12.8 | 1.4 | 0.4 | 3.3 | 3.6 | 0.7 | 3 | 13.6 | 14.2 | 14 | 6.7 |