Long-context understanding on LongBench (F1 Metric)
34F1 ScoreFull cache
Evaluation results
| Method | Links | |
|---|---|---|
| Full cacheModel=Qwen3.5-27B, Capacity (C)=Full2026.05 | 34 | |
| Full cacheModel=Qwen2.5-3B, Capacity (C)=Full2026.05 | 31.5 | |
| Full cacheModel=Qwen2.5-7B, Capacity (C)=Full2026.05 | 31.3 | |
| H2OModel=Qwen3.5-27B, Capacity (C)=512, Protection=+Prot2026.05 | 31 | |
| SnapKVModel=Qwen3.5-27B, Capacity (C)=512, Protection=+Prot2026.05 | 31 | |
| Full cacheModel=Qwen2.5-1.5B, Capacity (C)=Full2026.05 | 30.4 | |
| H2OModel=Qwen2.5-3B, Capacity (C)=512, Protection=+Prot2026.05 | 29.8 | |
| SnapKVModel=Qwen2.5-3B, Capacity (C)=512, Protection=+Prot2026.05 | 29.8 | |
| Ada-KVModel=Qwen2.5-3B, Capacity (C)=512, Protection=+Prot2026.05 | 29.8 | |
| QUESTModel=Qwen2.5-3B, Capacity (C)=512, Protection=+Prot2026.05 | 29.8 | |
| H2OModel=Qwen2.5-3B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 29 | |
| SnapKVModel=Qwen2.5-3B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 29 | |
| Ada-KVModel=Qwen2.5-3B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 29 | |
| QUESTModel=Qwen2.5-3B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 29 | |
| RandomModel=Qwen2.5-3B, Capacity (C)=512, Protection=+Prot2026.05 | 28.9 | |
| LRUModel=Qwen2.5-3B, Capacity (C)=512, Protection=+Prot2026.05 | 28.5 | |
| LRUModel=Qwen3.5-27B, Capacity (C)=512, Protection=+Prot2026.05 | 28.5 | |
| LRUModel=Qwen2.5-3B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 28.2 | |
| SnapKVModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 28.2 | |
| H2OModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 28.1 | |
| H2OModel=Qwen2.5-7B, Capacity (C)=512, Protection=+Prot2026.05 | 28 | |
| SnapKVModel=Qwen2.5-7B, Capacity (C)=512, Protection=+Prot2026.05 | 28 | |
| RandomModel=Qwen3.5-27B, Capacity (C)=512, Protection=+Prot2026.05 | 28 | |
| LRUModel=Qwen2.5-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 27.9 | |
| LRUModel=Qwen2.5-7B, Capacity (C)=512, Protection=+Prot2026.05 | 27.3 | |
| H2OModel=Qwen2.5-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 26.8 | |
| SnapKVModel=Qwen2.5-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 26.8 | |
| LRUModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 26.1 | |
| RandomModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 25.4 | |
| H2OModel=Qwen2.5-1.5B, Capacity (C)=512, Protection=+Prot2026.05 | 25.4 | |
| SnapKVModel=Qwen2.5-1.5B, Capacity (C)=512, Protection=+Prot2026.05 | 25.4 | |
| LRUModel=Qwen2.5-1.5B, Capacity (C)=512, Protection=+Prot2026.05 | 25.1 | |
| RandomModel=Qwen2.5-3B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 25 | |
| LRUModel=Qwen2.5-1.5B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 24.1 | |
| H2OModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=No prot2026.05 | 24 | |
| SnapKVModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=No prot2026.05 | 23.9 | |
| Full cacheModel=Mistral-7B, Capacity (C)=Full2026.05 | 23.4 | |
| H2OModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 23.2 | |
| SnapKVModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 23.2 | |
| H2OModel=Qwen2.5-1.5B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 23.1 | |
| SnapKVModel=Qwen2.5-1.5B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 23.1 | |
| H2OModel=Qwen2.5-3B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 23 | |
| SnapKVModel=Qwen2.5-3B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 23 | |
| Ada-KVModel=Qwen2.5-3B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 23 | |
| QUESTModel=Qwen2.5-3B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 23 | |
| LRUModel=Qwen2.5-3B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 22.9 | |
| H2OModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=No prot2026.05 | 22.6 | |
| SnapKVModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=No prot2026.05 | 22.6 | |
| RandomModel=Qwen2.5-3B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 22.5 | |
| Ref. ceilingModel=Yi-1.5-6B, Capacity (C)=256 (13%)2026.05 | 22.3 | |
| RandomModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=No prot2026.05 | 21.7 | |
| H2OModel=Mistral-7B, Capacity (C)=512, Protection=+Prot2026.05 | 21.5 | |
| LRUModel=Qwen2.5-1.5B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 21.4 | |
| SnapKVModel=Mistral-7B, Capacity (C)=512, Protection=+Prot2026.05 | 21.4 | |
| RandomModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 21.3 | |
| Ada-KVModel=Mistral-7B, Capacity (C)=512, Protection=+Prot2026.05 | 21.3 | |
| QUESTModel=Mistral-7B, Capacity (C)=512, Protection=+Prot2026.05 | 21.1 | |
| H2OModel=Qwen2.5-1.5B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 21 | |
| SnapKVModel=Qwen2.5-1.5B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 21 | |
| LRUModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 20.5 | |
| H2OModel=Qwen2.5-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 20.2 | |
| SnapKVModel=Qwen2.5-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 20.2 | |
| QUESTModel=Mistral-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 20.2 | |
| H2OModel=Mistral-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 20.1 | |
| Ada-KVModel=Mistral-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 20 | |
| SnapKVModel=Mistral-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 19.9 | |
| LRUModel=Mistral-7B, Capacity (C)=512, Protection=+Prot2026.05 | 19.4 | |
| LRUModel=Qwen2.5-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 19.2 | |
| LRUModel=Qwen3.5-27B, Capacity (C)=256 (13%), Protection=No prot2026.05 | 19 | |
| LRUModel=Mistral-7B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 18.8 | |
| Full cacheModel=Phi-3.5, Capacity (C)=Full2026.05 | 17.9 | |
| H2OModel=Yi-1.5-6B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 17.8 | |
| SnapKVModel=Yi-1.5-6B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 17.7 | |
| SnapKVModel=Phi-3.5, Capacity (C)=512, Protection=+Prot2026.05 | 17.1 | |
| H2OModel=Phi-3.5, Capacity (C)=512, Protection=+Prot2026.05 | 17 | |
| Ada-KVModel=Phi-3.5, Capacity (C)=512, Protection=+Prot2026.05 | 17 | |
| QUESTModel=Phi-3.5, Capacity (C)=512, Protection=+Prot2026.05 | 17 | |
| QUESTModel=Mistral-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 16.8 | |
| H2OModel=Mistral-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 16.7 | |
| SnapKVModel=Mistral-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 16.6 | |
| Ada-KVModel=Mistral-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 16.6 | |
| LRUModel=Mistral-7B, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 16.5 | |
| RandomModel=Yi-1.5-6B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 16.3 | |
| LRUModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=No prot2026.05 | 16 | |
| LRUModel=Phi-3.5, Capacity (C)=512, Protection=+Prot2026.05 | 16 | |
| RandomModel=Qwen3.5-27B, Capacity (C)=128 (7%), Protection=No prot2026.05 | 15.7 | |
| LRUModel=Yi-1.5-6B, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 15.3 | |
| H2OModel=Phi-3.5, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 14.3 | |
| SnapKVModel=Phi-3.5, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 14.2 | |
| LRUModel=Phi-3.5, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 14.1 | |
| Ada-KVModel=Phi-3.5, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 14.1 | |
| QUESTModel=Phi-3.5, Capacity (C)=256 (13%), Protection=+Prot2026.05 | 14 | |
| LRUModel=Phi-3.5, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 12.6 | |
| H2OModel=Phi-3.5, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 11.5 | |
| QUESTModel=Phi-3.5, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 11.5 | |
| SnapKVModel=Phi-3.5, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 11.4 | |
| Ada-KVModel=Phi-3.5, Capacity (C)=128 (7%), Protection=+Prot2026.05 | 11.4 | |
| H2OModel=Phi-3.5, Capacity (C)=256 (13%), Protection=No prot2026.05 | 6.4 | |
| QUESTModel=Phi-3.5, Capacity (C)=256 (13%), Protection=No prot2026.05 | 6.3 | |
| H2OModel=Mistral-7B, Capacity (C)=256 (13%), Protection=No prot2026.05 | 6 |