Long-context Language Modeling on LongBench (Generation Score, Attention Recall)
50.6Generation ScoreRaBitQCache
Evaluation results
| Method | Links | ||
|---|---|---|---|
| RaBitQCacheBudget=p = 0.95 (17.33%), Model=LLaMA-3.1-8B2026.06 | 50.6 | 90.7 | |
| FullBudget=-(100%), Model=LLaMA-3.1-8B2026.06 | 50.58 | 100 | |
| Double SparseBudget=8192 (64.60%), Model=LLaMA-3.1-8B2026.06 | 50.47 | 96.9 | |
| QuestBudget=8192 (64.57%), Model=LLaMA-3.1-8B2026.06 | 50.43 | 97 | |
| Double SparseBudget=4096 (40.33%), Model=LLaMA-3.1-8B2026.06 | 50.4 | 90.7 | |
| OracleBudget=1024 (11.4%), Model=LLaMA-3.1-8B2026.06 | 50.31 | 86.9 | |
| Double SparseBudget=1024 (11.42%), Model=LLaMA-3.1-8B2026.06 | 50.28 | 68.4 | |
| SparQBudget=Ratio=0.25 (25.00%), Model=LLaMA-3.1-8B2026.06 | 50.15 | 89.8 | |
| QuestBudget=4096 (40.29%), Model=LLaMA-3.1-8B2026.06 | 49.24 | 92.9 | |
| Double SparseBudget=256 (2.86%), Model=LLaMA-3.1-8B2026.06 | 49.06 | 42.7 | |
| QuestBudget=1024 (11.38%), Model=LLaMA-3.1-8B2026.06 | 46.52 | 82.6 | |
| QuestBudget=256 (2.84%), Model=LLaMA-3.1-8B2026.06 | 36.71 | 53.3 | |
| QuestBudget=4096 (40.29%), Model=Longchat-7B-v1.5-32k2026.06 | 36.28 | 86.1 | |
| RaBitQCacheBudget=p = 0.95 (17.33%), Model=Longchat-7B-v1.5-32k2026.06 | 36.21 | 89.8 | |
| QuestBudget=8192 (64.57%), Model=Longchat-7B-v1.5-32k2026.06 | 36.11 | 94.7 | |
| SparQBudget=Ratio=0.25 (25.00%), Model=Longchat-7B-v1.5-32k2026.06 | 35.91 | 93.1 | |
| OracleBudget=1024 (11.4%), Model=Longchat-7B-v1.5-32k2026.06 | 35.84 | 89.3 | |
| FullBudget=-(100%), Model=Longchat-7B-v1.5-32k2026.06 | 35.78 | 100 | |
| QuestBudget=1024 (11.38%), Model=Longchat-7B-v1.5-32k2026.06 | 35.3 | 58.1 | |
| Double SparseBudget=8192 (64.60%), Model=Longchat-7B-v1.5-32k2026.06 | 32.58 | 67.4 | |
| QuestBudget=256 (2.84%), Model=Longchat-7B-v1.5-32k2026.06 | 30.5 | 27.3 | |
| Double SparseBudget=4096 (40.33%), Model=Longchat-7B-v1.5-32k2026.06 | 21.28 | 47.8 | |
| Double SparseBudget=1024 (11.42%), Model=Longchat-7B-v1.5-32k2026.06 | 9.52 | 20.4 | |
| Double SparseBudget=256 (2.86%), Model=Longchat-7B-v1.5-32k2026.06 | 3.75 | 10 |