Variable Tracking on RULER-VT
99.9AccuracyFP8
Evaluation Results
| Method | Links | |
|---|---|---|
| FP8Backbone=Llama 3.1 8B, Compression Ratio (CR)=2, Evaluation Protocol=1-shot2025.11 | 99.9 | |
| VanillaBackbone=Llama 3.1 8B, Compression Ratio (CR)=1, Evaluation Protocol=1-shot2025.11 | 99.8 | |
| GEAR2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=1-shot2025.11 | 99.8 | |
| xKVBackbone=Llama 3.1 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=1-shot2025.11 | 99.8 | |
| VanillaBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1, Evaluation Protocol=1-shot2025.11 | 99.8 | |
| FP8Backbone=Mistral NeMo 12B, Compression Ratio (CR)=2, Evaluation Protocol=1-shot2025.11 | 99.8 | |
| TOVABackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=1-shot2025.11 | 99.7 | |
| TOVABackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=1-shot2025.11 | 99.6 | |
| kvtc8×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=10-11, Evaluation Protocol=1-shot2025.11 | 99.5 | |
| kvtc16×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=17-20, Evaluation Protocol=1-shot2025.11 | 99.5 | |
| VanillaBackbone=MN-Minitron 8B, Compression Ratio (CR)=1, Evaluation Protocol=1-shot2025.11 | 99.4 | |
| GEAR2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=1-shot2025.11 | 99.4 | |
| xKVBackbone=Mistral NeMo 12B, Compression Ratio (CR)=1-5, Evaluation Protocol=1-shot2025.11 | 99.4 | |
| TOVABackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=1-shot2025.11 | 99.3 | |
| FP8Backbone=MN-Minitron 8B, Compression Ratio (CR)=2, Evaluation Protocol=1-shot2025.11 | 99.2 | |
| kvtc8×Backbone=Llama 3.1 8B, Compression Ratio (CR)=9-10, Evaluation Protocol=1-shot2025.11 | 99.1 | |
| kvtc16×Backbone=Llama 3.1 8B, Compression Ratio (CR)=18-22, Evaluation Protocol=1-shot2025.11 | 99.1 | |
| xKVBackbone=MN-Minitron 8B, Compression Ratio (CR)=1-5, Evaluation Protocol=1-shot2025.11 | 99.1 | |
| KIVI2-bitBackbone=Llama 3.1 8B, Compression Ratio (CR)=5, Evaluation Protocol=1-shot2025.11 | 98.9 | |
| kvtc32×Backbone=Llama 3.1 8B, Compression Ratio (CR)=34-44, Evaluation Protocol=1-shot2025.11 | 98.9 | |
| kvtc32×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=31-43, Evaluation Protocol=1-shot2025.11 | 98.9 | |
| kvtc8×Backbone=MN-Minitron 8B, Compression Ratio (CR)=10-11, Evaluation Protocol=1-shot2025.11 | 98.8 | |
| kvtc16×Backbone=MN-Minitron 8B, Compression Ratio (CR)=17-21, Evaluation Protocol=1-shot2025.11 | 98.8 | |
| GEAR2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=1-shot2025.11 | 98.3 | |
| KIVI2-bitBackbone=Mistral NeMo 12B, Compression Ratio (CR)=5, Evaluation Protocol=1-shot2025.11 | 98.3 | |
| kvtc64×Backbone=Mistral NeMo 12B, Compression Ratio (CR)=51-87, Evaluation Protocol=1-shot2025.11 | 98 | |
| KIVI2-bitBackbone=MN-Minitron 8B, Compression Ratio (CR)=5, Evaluation Protocol=1-shot2025.11 | 96.8 | |
| kvtc32×Backbone=MN-Minitron 8B, Compression Ratio (CR)=32-46, Evaluation Protocol=1-shot2025.11 | 96 | |
| kvtc64×Backbone=Llama 3.1 8B, Compression Ratio (CR)=60-88, Evaluation Protocol=1-shot2025.11 | 95.9 | |
| kvtc64×Backbone=MN-Minitron 8B, Compression Ratio (CR)=53-95, Evaluation Protocol=1-shot2025.11 | 93.4 | |
| H2OBackbone=Llama 3.1 8B, Compression Ratio (CR)=8, Evaluation Protocol=1-shot2025.11 | 50.4 | |
| H2OBackbone=MN-Minitron 8B, Compression Ratio (CR)=8, Evaluation Protocol=1-shot2025.11 | 39.2 | |
| H2OBackbone=Mistral NeMo 12B, Compression Ratio (CR)=8, Evaluation Protocol=1-shot2025.11 | 35.2 |