Question Answering on NarrativeQA LongBench (32K Context)
17.2F1 ScoreFull cache
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full cacheModel=Qwen2.5-3B, Condition=Full cache, C (Cache capacity)=32768, Retention=100%2026.05 | 17.2 | 100 | — | |
| LRU + protModel=Qwen2.5-3B, Condition=LRU + prot, C (Cache capacity)=4096, Retention=12.5%2026.05 | 10.8 | 62.8 | — | |
| Random + protModel=Qwen2.5-3B, Condition=Random + prot, C (Cache capacity)=4096, Retention=12.5%2026.05 | 10.3 | 59.9 | 0.873 | |
| LRU + protModel=Qwen2.5-7B, Condition=LRU + prot, C (Cache capacity)=4096, Retention=12.5%2026.05 | 8.6 | 112.1 | — | |
| H2O + protModel=Qwen2.5-7B, Condition=H2O + prot, C (Cache capacity)=4096, Retention=12.5%2026.05 | 8.6 | 111.9 | 0.893 | |
| SnapKV + protModel=Qwen2.5-7B, Condition=SnapKV + prot, C (Cache capacity)=4096, Retention=12.5%2026.05 | 8.6 | 111.9 | 0.893 | |
| Random + protModel=Qwen2.5-7B, Condition=Random + prot, C (Cache capacity)=4096, Retention=12.5%2026.05 | 8.3 | 107.2 | 0.99 | |
| Full cacheModel=Qwen2.5-7B, Condition=Full cache, C (Cache capacity)=32768, Retention=100%2026.05 | 7.7 | 100 | — | |
| Random (no prot)Model=Qwen2.5-7B, Condition=Random (no prot), C (Cache capacity)=4096, Retention=12.5%2026.05 | 1.7 | 22.4 | 0.001 | |
| H2O (no prot)Model=Qwen2.5-7B, Condition=H2O (no prot), C (Cache capacity)=4096, Retention=12.5%2026.05 | 1.7 | 21.9 | 0.001 | |
| LRU (no prot)Model=Qwen2.5-7B, Condition=LRU (no prot), C (Cache capacity)=4096, Retention=12.5%2026.05 | 1.3 | 17.5 | 0.001 | |
| LRU (no prot)Model=Qwen2.5-3B, Condition=LRU (no prot), C (Cache capacity)=4096, Retention=12.5%2026.05 | 0.8 | 4.6 | 0.001 |