Key-value Retrieval on LITM (Lost in the Middle)
99.9AccuracyxKV
Evaluation Results
| Method | Links | |
|---|---|---|
| xKVBackbone=Llama 3.1 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=0-shot2025.11 | 99.9 | |
| kvtc8×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=10-11, Evaluation Protocol=0-shot2025.11 | 99.9 | |
| VanillaBackbone=MN-Minitron 8B, Compression Ratio (CR)=1, Evaluation Protocol=0-shot2025.11 | 99.8 | |
| FP8Backbone=MN-Minitron 8B, Compression Ratio (CR)=2, Evaluation Protocol=0-shot2025.11 | 99.8 | |
| kvtc16×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=17-20, Evaluation Protocol=0-shot2025.11 | 99.8 | |
| xKVBackbone=MN-Minitron 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=0-shot2025.11 | 99.6 | |
| kvtc32×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=31-43, Evaluation Protocol=0-shot2025.11 | 99.6 | |
| VanillaBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1, Evaluation Protocol=0-shot2025.11 | 99.5 | |
| VanillaBackbone=Llama 3.1 8B, Compression Ratio (CR)=1, Evaluation Protocol=0-shot2025.11 | 99.4 | |
| FP8Backbone=Llama 3.1 8B, Compression Ratio (CR)=2, Evaluation Protocol=0-shot2025.11 | 99.4 | |
| kvtc8×Backbone=MN-Minitron 8B, Compression Ratio (CR)=10-11, Evaluation Protocol=0-shot2025.11 | 99.4 | |
| kvtc8×Backbone=Llama 3.1 8B, Compression Ratio (CR)=9-10, Evaluation Protocol=0-shot2025.11 | 99.3 | |
| kvtc16×Backbone=Llama 3.1 8B, Compression Ratio (CR)=18-22, Evaluation Protocol=0-shot2025.11 | 99.3 | |
| kvtc16×Backbone=MN-Minitron 8B, Compression Ratio (CR)=17-21, Evaluation Protocol=0-shot2025.11 | 99.3 | |
| kvtc32×Backbone=Llama 3.1 8B, Compression Ratio (CR)=34-44, Evaluation Protocol=0-shot2025.11 | 99.1 | |
| FP8Backbone=Mistral NeMo 12B, Compression Ratio (CR)=2, Evaluation Protocol=0-shot2025.11 | 99 | |
| xKVBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1-5, Evaluation Protocol=0-shot2025.11 | 97.9 | |
| GEAR2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.11 | 96.9 | |
| GEAR2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.11 | 96.9 | |
| GEAR2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.11 | 96 | |
| kvtc64×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=51-87, Evaluation Protocol=0-shot2025.11 | 95.3 | |
| KIVI2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.11 | 91.9 | |
| kvtc64×Backbone=Llama 3.1 8B, Compression Ratio (CR)=60-88, Evaluation Protocol=0-shot2025.11 | 90.2 | |
| KIVI2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.11 | 88.8 | |
| kvtc32×Backbone=MN-Minitron 8B, Compression Ratio (CR)=32-46, Evaluation Protocol=0-shot2025.11 | 86.9 | |
| KIVI2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.11 | 86.3 | |
| kvtc64×Backbone=MN-Minitron 8B, Compression Ratio (CR)=53-95, Evaluation Protocol=0-shot2025.11 | 59.5 | |
| H2OBackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.11 | 20.2 | |
| H2OBackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.11 | 16.6 | |
| H2OBackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.11 | 16.2 | |
| TOVABackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.11 | 8.7 | |
| TOVABackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.11 | 1.2 | |
| TOVABackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.11 | 0.3 |