Long-context Understanding on RULER 4k (test)
95.7RULER 4k ScoreExpectedAttention
Evaluation results
| Method | Links | |
|---|---|---|
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 95.7 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 95.6 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 95.6 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 95.5 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=0%2026.05 | 95.3 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=0%2026.05 | 95.3 | |
| TOVAModel=Qwen3-8B, Compression Ratio=0%2026.05 | 95.3 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=0%2026.05 | 95.3 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=0%2026.05 | 95.3 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 95.3 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 95.3 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 95.3 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 95.3 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 95.3 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 95.3 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=10%2026.05 | 93.8 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=10%2026.05 | 93.8 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 93.3 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 93.2 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=10%2026.05 | 92.6 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 92.2 | |
| IndexMemModel=Mistral-7B, Compression Ratio=0%2026.05 | 92.1 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=0%2026.05 | 92.1 | |
| TOVAModel=Mistral-7B, Compression Ratio=0%2026.05 | 92.1 | |
| SnapKVModel=Mistral-7B, Compression Ratio=0%2026.05 | 92.1 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=0%2026.05 | 92.1 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=10%2026.05 | 91.4 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=25%2026.05 | 90.3 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=25%2026.05 | 89.4 | |
| TOVAModel=Qwen3-8B, Compression Ratio=10%2026.05 | 89 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 88.8 | |
| IndexMemModel=Mistral-7B, Compression Ratio=10%2026.05 | 88.6 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 87.3 | |
| IndexMemModel=Mistral-7B, Compression Ratio=25%2026.05 | 85 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 84.5 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=25%2026.05 | 84 | |
| TOVAModel=Mistral-7B, Compression Ratio=10%2026.05 | 83.1 | |
| IndexMemModel=Mistral-7B, Compression Ratio=50%2026.05 | 82.8 | |
| TOVAModel=Qwen3-8B, Compression Ratio=25%2026.05 | 82.5 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 81.8 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 79.4 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=50%2026.05 | 78.9 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=50%2026.05 | 78.6 | |
| TOVAModel=Mistral-7B, Compression Ratio=25%2026.05 | 78.4 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 78.3 | |
| TOVAModel=Qwen3-8B, Compression Ratio=50%2026.05 | 77.6 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=10%2026.05 | 77.1 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 76.2 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 75.9 | |
| TOVAModel=Mistral-7B, Compression Ratio=50%2026.05 | 75.5 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=25%2026.05 | 75.1 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=75%2026.05 | 72.9 | |
| SnapKVModel=Mistral-7B, Compression Ratio=10%2026.05 | 71.5 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=10%2026.05 | 70.8 | |
| IndexMemModel=Mistral-7B, Compression Ratio=75%2026.05 | 68.8 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=25%2026.05 | 66.3 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 65.8 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=75%2026.05 | 64.4 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 63.3 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 63.2 | |
| TOVAModel=Qwen3-8B, Compression Ratio=75%2026.05 | 62.4 | |
| TOVAModel=Mistral-7B, Compression Ratio=75%2026.05 | 62 | |
| SnapKVModel=Mistral-7B, Compression Ratio=25%2026.05 | 57.6 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=25%2026.05 | 57.1 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=50%2026.05 | 55.7 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 55.2 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=90%2026.05 | 46.2 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=50%2026.05 | 44.8 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 43.6 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 43.4 | |
| IndexMemModel=Mistral-7B, Compression Ratio=90%2026.05 | 41.5 | |
| SnapKVModel=Mistral-7B, Compression Ratio=50%2026.05 | 41.3 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=50%2026.05 | 40.8 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=50%2026.05 | 38.6 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=90%2026.05 | 37.9 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 37.5 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=75%2026.05 | 33.1 | |
| SnapKVModel=Mistral-7B, Compression Ratio=75%2026.05 | 31.9 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 30.6 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=75%2026.05 | 29.7 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=75%2026.05 | 29.6 | |
| TOVAModel=Mistral-7B, Compression Ratio=90%2026.05 | 29.2 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=75%2026.05 | 28.1 | |
| TOVAModel=Qwen3-8B, Compression Ratio=90%2026.05 | 24.7 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 23.1 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=90%2026.05 | 19.9 | |
| SnapKVModel=Mistral-7B, Compression Ratio=90%2026.05 | 19.8 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=90%2026.05 | 19.5 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=90%2026.05 | 19.2 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=90%2026.05 | 13.2 |