List your company
ResearchFormerly WizWand

Key-value Retrieval on LITM (Lost in the Middle)

99.9AccuracyxKV
-3.68423.20850.176.992Nov 3, 2025
Updated 7mo ago

Evaluation results

MethodLinks
xKVBackbone=Llama 3.1 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=0-shot2025.1199.9
kvtc8×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=10-11, Evaluation Protocol=0-shot2025.1199.9
VanillaBackbone=MN-Minitron 8B, Compression Ratio (CR)=1, Evaluation Protocol=0-shot2025.1199.8
FP8Backbone=MN-Minitron 8B, Compression Ratio (CR)=2, Evaluation Protocol=0-shot2025.1199.8
kvtc16×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=17-20, Evaluation Protocol=0-shot2025.1199.8
xKVBackbone=MN-Minitron 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=0-shot2025.1199.6
kvtc32×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=31-43, Evaluation Protocol=0-shot2025.1199.6
VanillaBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1, Evaluation Protocol=0-shot2025.1199.5
VanillaBackbone=Llama 3.1 8B, Compression Ratio (CR)=1, Evaluation Protocol=0-shot2025.1199.4
FP8Backbone=Llama 3.1 8B, Compression Ratio (CR)=2, Evaluation Protocol=0-shot2025.1199.4
kvtc8×Backbone=MN-Minitron 8B, Compression Ratio (CR)=10-11, Evaluation Protocol=0-shot2025.1199.4
kvtc8×Backbone=Llama 3.1 8B, Compression Ratio (CR)=9-10, Evaluation Protocol=0-shot2025.1199.3
kvtc16×Backbone=Llama 3.1 8B, Compression Ratio (CR)=18-22, Evaluation Protocol=0-shot2025.1199.3
kvtc16×Backbone=MN-Minitron 8B, Compression Ratio (CR)=17-21, Evaluation Protocol=0-shot2025.1199.3
kvtc32×Backbone=Llama 3.1 8B, Compression Ratio (CR)=34-44, Evaluation Protocol=0-shot2025.1199.1
FP8Backbone=Mistral NeMo 12B, Compression Ratio (CR)=2, Evaluation Protocol=0-shot2025.1199
xKVBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1-5, Evaluation Protocol=0-shot2025.1197.9
GEAR2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.1196.9
GEAR2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.1196.9
GEAR2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.1196
kvtc64×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=51-87, Evaluation Protocol=0-shot2025.1195.3
KIVI2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.1191.9
kvtc64×Backbone=Llama 3.1 8B, Compression Ratio (CR)=60-88, Evaluation Protocol=0-shot2025.1190.2
KIVI2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.1188.8
kvtc32×Backbone=MN-Minitron 8B, Compression Ratio (CR)=32-46, Evaluation Protocol=0-shot2025.1186.9
KIVI2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=0-shot2025.1186.3
kvtc64×Backbone=MN-Minitron 8B, Compression Ratio (CR)=53-95, Evaluation Protocol=0-shot2025.1159.5
H2OBackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.1120.2
H2OBackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.1116.6
H2OBackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.1116.2
TOVABackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.118.7
TOVABackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.111.2
TOVABackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=0-shot2025.110.3