Language Understanding on MMLU-M
27.4AccuracyZipCal
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ZipCalModel=gemma-2-9b-it, Compression Method=Wanda, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 27.4 | — | 4.89 | |
| ZipCalModel=gemma-2-9b-it, Compression Method=GPTQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 26.29 | — | 3.85 | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=Mean, Quantization=GPTQ W4A162026.03 | 26.27 | 10.7 | 3.83 | |
| DenseModel=Llama-3.1-8B-Instruct, Quantization=Full Precision2026.03 | 24.44 | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Model State=Dense2026.03 | 24.44 | — | — | |
| DenseModel=Llama-3.1-8B-Instruct, Compression Method=Dense2026.03 | 24.44 | — | — | |
| COLAModel=Llama-3.1-8B-Instruct, Compression Method=AWQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 24.44 | — | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Compression Method=AWQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 24.44 | — | 0 | |
| ZipCalModel=Llama-3.1-8B-Instruct, Compression Method=Wanda, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 24 | — | 0.81 | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=LangMod, Quantization=GPTQ W4A162026.03 | 23.33 | 5,400 | — | |
| ZipCalModel=gemma-2-9b-it, Compression Method=2SSP, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 23.33 | — | 4.07 | |
| COLAModel=Llama-3.1-8B-Instruct, Compression Method=Wanda, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 23.19 | — | — | |
| COLAModel=Llama-3.1-8B-Instruct, Compression Method=GPTQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 23.19 | — | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=Mean, Quantization=GPTQ W4A162026.03 | 23.11 | 10.7 | -0.08 | |
| ZipCalModel=Llama-3.1-8B-Instruct, Compression Method=GPTQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 23.01 | — | -0.18 | |
| ZipCalModel=Llama-3.1-8B-Instruct, Compression Method=2SSP, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 22.54 | — | 0.32 | |
| COLAModel=gemma-2-9b-it, Compression Method=Wanda, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 22.51 | — | — | |
| COLAModel=gemma-2-9b-it, Compression Method=GPTQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 22.44 | — | — | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=Mean, Runtime=2443s2026.03 | 22.22 | — | — | |
| COLAModel=Llama-3.1-8B-Instruct, Compression Method=2SSP, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 22.22 | — | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=Mean, Runtime=10.7s2026.03 | 21.96 | — | -0.26 | |
| DenseModel=gemma-2-9b-it, Quantization=Full Precision2026.03 | 21.48 | — | — | |
| DenseModel=gemma-2-9b-it, Model State=Dense2026.03 | 21.48 | — | — | |
| DenseModel=gemma-2-9b-it, Compression Method=Dense2026.03 | 21.48 | — | — | |
| COLAModel=gemma-2-9b-it, Compression Method=AWQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 21.4 | — | — | |
| ZipCalModel=gemma-2-9b-it, Compression Method=AWQ, Compression Setting (Sparsity/Bit-width)=W4A162026.03 | 21.3 | — | -0.1 | |
| COLAModel=gemma-2-9b-it, Calibration Category=Mean, Runtime=2790s2026.03 | 19.26 | — | — | |
| COLAModel=gemma-2-9b-it, Compression Method=2SSP, Compression Setting (Sparsity/Bit-width)=25%2026.03 | 19.26 | — | — | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=Mean, Runtime=10.7s2026.03 | 19.02 | — | -0.24 |