Long-context Understanding on RULER 16k (test)
93.5RULER ScoreIndexMem
Evaluation results
| Method | Links | |
|---|---|---|
| IndexMemModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 93.5 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 93.4 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 93.4 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 93.4 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 93.4 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 93.4 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=0%2026.05 | 93.4 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=0%2026.05 | 92.9 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=0%2026.05 | 92.9 | |
| TOVAModel=Qwen3-8B, Compression Ratio=0%2026.05 | 92.9 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=0%2026.05 | 92.9 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=0%2026.05 | 92.9 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 92.9 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 92.8 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=10%2026.05 | 91.6 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 90.9 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=10%2026.05 | 90.1 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 89.4 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=10%2026.05 | 89.2 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=10%2026.05 | 88.9 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 88.7 | |
| TOVAModel=Qwen3-8B, Compression Ratio=10%2026.05 | 88.3 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=25%2026.05 | 86.8 | |
| IndexMemModel=Mistral-7B, Compression Ratio=0%2026.05 | 86.1 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=0%2026.05 | 86.1 | |
| TOVAModel=Mistral-7B, Compression Ratio=0%2026.05 | 86.1 | |
| SnapKVModel=Mistral-7B, Compression Ratio=0%2026.05 | 86.1 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=0%2026.05 | 86.1 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 86.1 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 86 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=10%2026.05 | 85.2 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=25%2026.05 | 82.9 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 82 | |
| TOVAModel=Qwen3-8B, Compression Ratio=25%2026.05 | 81.7 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=25%2026.05 | 81.5 | |
| IndexMemModel=Mistral-7B, Compression Ratio=10%2026.05 | 81 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=50%2026.05 | 80 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 77.9 | |
| TOVAModel=Mistral-7B, Compression Ratio=10%2026.05 | 77.7 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=25%2026.05 | 77.5 | |
| TOVAModel=Qwen3-8B, Compression Ratio=50%2026.05 | 76.2 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=25%2026.05 | 76.1 | |
| IndexMemModel=Mistral-7B, Compression Ratio=25%2026.05 | 75.9 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=50%2026.05 | 74.5 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 74.3 | |
| TOVAModel=Mistral-7B, Compression Ratio=25%2026.05 | 73.8 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=75%2026.05 | 72.2 | |
| TOVAModel=Mistral-7B, Compression Ratio=50%2026.05 | 70.1 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=10%2026.05 | 69.5 | |
| TOVAModel=Qwen3-8B, Compression Ratio=75%2026.05 | 68.7 | |
| IndexMemModel=Mistral-7B, Compression Ratio=50%2026.05 | 68.5 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 68.4 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 68 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=75%2026.05 | 66.9 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 66.4 | |
| IndexMemModel=Mistral-7B, Compression Ratio=75%2026.05 | 66 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=50%2026.05 | 62.8 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=50%2026.05 | 62.5 | |
| SnapKVModel=Mistral-7B, Compression Ratio=10%2026.05 | 62.3 | |
| TOVAModel=Mistral-7B, Compression Ratio=75%2026.05 | 61 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=25%2026.05 | 59.3 | |
| TOVAModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 59.2 | |
| IndexMemModel=Qwen3-8B, Compression Ratio=90%2026.05 | 56 | |
| IndexMemModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 56 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=10%2026.05 | 53.4 | |
| KeyDiffModel=Qwen3-8B, Compression Ratio=90%2026.05 | 53.1 | |
| IndexMemModel=Mistral-7B, Compression Ratio=90%2026.05 | 52.9 | |
| TOVAModel=Qwen3-8B, Compression Ratio=90%2026.05 | 52.4 | |
| SnapKVModel=Mistral-7B, Compression Ratio=25%2026.05 | 49.5 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=50%2026.05 | 47.8 | |
| TOVAModel=Mistral-7B, Compression Ratio=90%2026.05 | 47.7 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=25%2026.05 | 45 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 43.1 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=75%2026.05 | 41.7 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=50%2026.05 | 41 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=75%2026.05 | 39.6 | |
| SnapKVModel=Mistral-7B, Compression Ratio=50%2026.05 | 36.5 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=50%2026.05 | 33.2 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=75%2026.05 | 32.6 | |
| SnapKVModel=Mistral-7B, Compression Ratio=75%2026.05 | 27.1 | |
| SnapKVModel=Qwen3-8B, Compression Ratio=90%2026.05 | 26.8 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=75%2026.05 | 26.2 | |
| SnapKVModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 25.6 | |
| ExpectedAttentionModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 25.5 | |
| PyramidKVModel=Llama3.1-8B, Compression Ratio=90%2026.05 | 24.1 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=75%2026.05 | 22.9 | |
| PyramidKVModel=Qwen3-8B, Compression Ratio=90%2026.05 | 22 | |
| SnapKVModel=Mistral-7B, Compression Ratio=90%2026.05 | 21.3 | |
| PyramidKVModel=Mistral-7B, Compression Ratio=90%2026.05 | 18 | |
| ExpectedAttentionModel=Mistral-7B, Compression Ratio=90%2026.05 | 15.4 |