Multi-modal Long-context Benchmarking on MileBench
57.23Task T ScoreFlashCache
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| FlashCacheBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.12025.11 | 57.23 | 59.08 | 16.1 | 20.17 | |
| SnapKVBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.12025.11 | 57.06 | 58.71 | 16.06 | 18.83 | |
| StreamingLLMBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.12025.11 | 57 | 56.15 | 2.5 | 20 | |
| H2OBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.12025.11 | 56.94 | 58.33 | 2.5 | 21 | |
| Full CacheBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=1.02025.11 | 56.51 | 65.21 | 27.19 | 18.17 | |
| FlashCacheBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.052025.11 | 56.25 | 53.11 | 8.29 | 26.33 | |
| SnapKVBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.052025.11 | 55.69 | 49.07 | 6.25 | 19.83 | |
| FlashCacheBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.12025.11 | 55.65 | 60.93 | 18.45 | 14.5 | |
| Full CacheBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=1.02025.11 | 55.59 | 69.17 | 27.35 | 14.17 | |
| SnapKVBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.12025.11 | 55.58 | 61.3 | 7.19 | 15.5 | |
| LOOK-MBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.12025.11 | 55.08 | 58.54 | 5.78 | 13.67 | |
| StreamingLLMBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.12025.11 | 54.71 | 59.23 | 6.56 | 15.83 | |
| H2OBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.12025.11 | 54.58 | 58.65 | 6.25 | 14.83 | |
| FlashCacheBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.052025.11 | 54.55 | 56.19 | 10.63 | 14.83 | |
| MEDABackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.12025.11 | 54.24 | 57.05 | 5.47 | 13.67 | |
| H2OBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.052025.11 | 52.31 | 48.44 | 1.25 | 23.67 | |
| StreamingLLMBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.052025.11 | 51.94 | 42.34 | 1.25 | 26.83 | |
| LOOK-MBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.12025.11 | 51.91 | 58.99 | 2.81 | 24.67 | |
| MEDABackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.052025.11 | 51.86 | 50.01 | 2.65 | 26.33 | |
| MEDABackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.052025.11 | 51.74 | 52.95 | 5.16 | 15.33 | |
| LOOK-MBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.052025.11 | 51.69 | 53.67 | 5.31 | 16.33 | |
| SnapKVBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.052025.11 | 51.52 | 56.33 | 6.25 | 15.33 | |
| MEDABackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.12025.11 | 51.31 | 48.45 | 2.97 | 28.67 | |
| H2OBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.052025.11 | 50.47 | 52.77 | 5.31 | 11.33 | |
| LOOK-MBackbone=Qwen2.5-VL-32B, KV Cache retention ratio (rho)=0.052025.11 | 49.97 | 45.77 | 2.5 | 27.83 | |
| StreamingLLMBackbone=Qwen2.5-VL-7B, KV Cache retention ratio (rho)=0.052025.11 | 49.46 | 53.64 | 5.63 | 15 | |
| LOOK-MBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.052025.11 | 48.01 | 56.99 | 12.5 | 11.5 | |
| FlashCacheBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.12025.11 | 47.09 | 58.56 | 20 | 12.83 | |
| H2OBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.052025.11 | 47.01 | 56.99 | 12.5 | 11.5 | |
| Full CacheBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=1.02025.11 | 46.97 | 65.33 | 27.03 | 11.67 | |
| SnapKVBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.12025.11 | 46.97 | 57.62 | 10.47 | 12.67 | |
| H2OBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.12025.11 | 46.84 | 58.51 | 15.94 | 11.83 | |
| LOOK-MBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.12025.11 | 46.84 | 58.51 | 15.94 | 11.83 | |
| StreamingLLMBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.12025.11 | 46.81 | 58.4 | 6.72 | 12.5 | |
| StreamingLLMBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.052025.11 | 46.52 | 57.05 | 5.16 | 12 | |
| SnapKVBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.052025.11 | 45.97 | 57.82 | 7.97 | 13 | |
| FlashCacheBackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.052025.11 | 45.51 | 51.12 | 15.56 | 13.33 | |
| MEDABackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.12025.11 | 44.23 | 52.76 | 14.06 | 12.17 | |
| MEDABackbone=LLaVA-OneVision-1.5-8B, KV Cache retention ratio (rho)=0.052025.11 | 38.4 | 48.25 | 10.31 | 13 |