Question Answering on LongBench 48 items median 11K tokens
39.6F1 ScoreFull cache
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full cacheCache Capacity (C)=8192, Retention=100%, Model=Qwen2.5-3B2026.05 | 39.6 | 100 | — | |
| H2O + protCache Capacity (C)=1024, Retention=12.5%, Model=Qwen2.5-3B2026.05 | 34.5 | 87.2 | 0.445 | |
| SnapKV + protCache Capacity (C)=1024, Retention=12.5%, Model=Qwen2.5-3B2026.05 | 34.5 | 87.2 | 0.445 | |
| LRU + protCache Capacity (C)=1024, Retention=12.5%, Model=Qwen2.5-3B2026.05 | 33.6 | 85 | — | |
| LRU + protCache Capacity (C)=2048, Retention=25.0%, Model=Qwen2.5-3B2026.05 | 33.4 | 84.4 | — | |
| LRU + protCache Capacity (C)=4096, Retention=50.0%, Model=Qwen2.5-3B2026.05 | 31.4 | 79.4 | — | |
| Random + protCache Capacity (C)=4096, Retention=50.0%, Model=Qwen2.5-3B2026.05 | 30 | 75.8 | 0.673 | |
| LRU (no prot)Cache Capacity (C)=1024, Retention=12.5%, Model=Qwen2.5-3B2026.05 | 20.8 | 52.7 | 0.004 | |
| LRU (no prot)Cache Capacity (C)=2048, Retention=25.0%, Model=Qwen2.5-3B2026.05 | 18.7 | 47.3 | 0.002 | |
| LRU (no prot)Cache Capacity (C)=4096, Retention=50.0%, Model=Qwen2.5-3B2026.05 | 10.8 | 27.2 | 0.001 |