Long-context language understanding on RULER 0 shot v1 (test)
94.71CWE ScoreVanilla
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| VanillaModel=Llama 3.1 8B, Compression Ratio (CR)=12025.11 | 94.71 | 92.11 | 1,000 | 57.22 | 55.72 | 59.9 | |
| GEARModel=Llama 3.1 8B, Compression Ratio (CR)=52025.11 | 94.01 | 91.91 | 1,000 | 56.82 | 55.72 | 59.7 | |
| VanillaModel=Mistral NeMo 12B, Compression Ratio (CR)=12025.11 | 93.21 | 83.01 | 1,000 | 36.22 | 22.41 | 52.8 | |
| kvtc16xModel=Llama 3.1 8B, Compression Ratio (CR)=18-202025.11 | 92.41 | 89.01 | 1,000 | 57.22 | 55.22 | 59.2 | |
| VanillaModel=MN-Minitron 8B, Compression Ratio (CR)=12025.11 | 92.41 | 86.21 | 1,000 | 62.02 | 64.92 | 58.1 | |
| kvtc16xModel=Mistral NeMo 12B, Compression Ratio (CR)=192025.11 | 90.91 | 82.11 | 1,000 | 33.82 | 22.91 | 52.2 | |
| kvtc16xModel=MN-Minitron 8B, Compression Ratio (CR)=192025.11 | 85.31 | 86.31 | 1,000 | 62.22 | 64.72 | 57.3 |