Long-context retrieval on Multi-task NIAH 4K–128K context lengths Llama-3.1-8B-Instruct
100Accuracy (Single)fp16
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| fp16Backbone=Llama-3.1-8B-Instruct, Quantization Budget=fp162026.06 | 100 | 100 | 100 | 99.5 | 100 | 97.8 | 99.6 | |
| TQ-MSEBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K3V32026.06 | 100 | 99.5 | 100 | 100 | 99.5 | 87.4 | 97.7 | |
| Block-GTQBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K3V32026.06 | 100 | 100 | 100 | 100 | 99.7 | 90.7 | 98.4 | |
| TQ-MSEBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K3V22026.06 | 100 | 99 | 100 | 96.1 | 97 | 82 | 95.7 | |
| Block-GTQBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K3V22026.06 | 100 | 100 | 100 | 100 | 99.7 | 81.1 | 96.8 | |
| Block-GTQBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K2V22026.06 | 100 | 99 | 100 | 98.5 | 100 | 86.7 | 97.4 | |
| TQ-MSEBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K2V22026.06 | 93.9 | 70.7 | 80.8 | 47.6 | 71.9 | 58.8 | 70.6 | |
| KIVI-ScaleOnlyBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K3V22026.06 | 39.4 | 50 | 34.3 | 31 | 24.1 | 27.6 | 34.4 | |
| KIVI-ScaleOnlyBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K3V32026.06 | 38.4 | 49.5 | 36.4 | 33.3 | 26.6 | 28.3 | 35.4 | |
| KIVI-ScaleOnlyBackbone=Llama-3.1-8B-Instruct, Quantization Budget=K2V22026.06 | 30.8 | 32.8 | 23.2 | 20.9 | 16.2 | 21.6 | 24.2 |