Long-context language understanding on LongBench 1 host v1 (test)
46.232WQA Scorekvtc16x
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| kvtc16xModel=MN-Minitron 8B, Compression Ratio (CR)=192025.11 | 46.23 | 42.62 | 27.12 | 23.1 | 35.91 | 57.3 | |
| VanillaModel=MN-Minitron 8B, Compression Ratio (CR)=12025.11 | 45.83 | 42.62 | 27.42 | 23.4 | 36.11 | 58.1 | |
| kvtc16xModel=Mistral NeMo 12B, Compression Ratio (CR)=192025.11 | 43.73 | 51.02 | 26.62 | 25.2 | 45.91 | 52.2 | |
| VanillaModel=Mistral NeMo 12B, Compression Ratio (CR)=12025.11 | 43.33 | 51.52 | 27.22 | 25.7 | 45.61 | 52.8 | |
| KIVIModel=Llama 3.1 8B, Compression Ratio (CR)=52025.11 | 42.13 | 50.12 | 31.73 | 25.7 | 45.81 | 58.8 | |
| TOVAModel=Llama 3.1 8B, Compression Ratio (CR)=82025.11 | 41.33 | 50.42 | 32.63 | 25.4 | 46.11 | 54.8 | |
| VanillaModel=Llama 3.1 8B, Compression Ratio (CR)=12025.11 | 40.83 | 50.32 | 33.83 | 26.9 | 47.31 | 59.9 | |
| GEARModel=Llama 3.1 8B, Compression Ratio (CR)=52025.11 | 40.73 | 49.62 | 33.83 | 27.2 | 47.21 | 59.7 | |
| FP8Model=Llama 3.1 8B, Compression Ratio (CR)=22025.11 | 40.63 | 50.92 | 33.83 | 26.1 | 47.11 | 59.9 | |
| kvtc64xModel=Llama 3.1 8B, Compression Ratio (CR)=62-782025.11 | 40.63 | 50.22 | 34.13 | 25.2 | 45.61 | 57.6 | |
| kvtc16xModel=Llama 3.1 8B, Compression Ratio (CR)=18-202025.11 | 40.33 | 51.12 | 33.73 | 26.4 | 47.01 | 59.2 | |
| kvtc32xModel=Llama 3.1 8B, Compression Ratio (CR)=35-392025.11 | 40.03 | 49.52 | 33.93 | 25.9 | 46.61 | 58.5 | |
| xKVModel=Llama 3.1 8B, Compression Ratio (CR)=4-62025.11 | 39.53 | 48.72 | 31.53 | 24.4 | 45.71 | 55.7 | |
| H2OModel=Llama 3.1 8B, Compression Ratio (CR)=82025.11 | 38.53 | 40.82 | 32.43 | 25.4 | 46.81 | 42 |