Multi-task Language Understanding on MMLU-M
28.79AccuracyZipCal
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ZipCalModel=gemma-2-9b-it, Calibration Category=NLI, Pruning Method=Wanda, Sparsity=25%2026.03 | 28.79 | — | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=LangMod, Pruning Method=Wanda, Sparsity=25%2026.03 | 27.4 | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=Math, Pruning Method=Wanda, Sparsity=25%2026.03 | 26.29 | — | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=CommQA, Pruning Method=Wanda, Sparsity=25%2026.03 | 26.29 | — | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=Mean, Pruning Method=Wanda, Sparsity=25%2026.03 | 26.27 | 3.76 | |
| DenseModel=Llama-3.1-8B-Instruct, Pruning Method=None, Sparsity=0%2026.03 | 24.44 | — | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=Math, Pruning Method=Wanda, Sparsity=25%2026.03 | 24.44 | — | |
| COLAModel=gemma-2-9b-it, Calibration Category=LangMod, Pruning Method=Wanda, Sparsity=25%2026.03 | 24.44 | — | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=Math, Pruning Method=Wanda, Sparsity=25%2026.03 | 24.44 | — | |
| ZipCalModel=gemma-2-9b-it, Calibration Category=KnowTran, Pruning Method=Wanda, Sparsity=25%2026.03 | 24.44 | — | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=LangMod, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.33 | — | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=KnowTran, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.33 | — | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=Mean, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.19 | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=Mean, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.11 | -0.08 | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=CommQA, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.01 | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=LangMod, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.01 | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=KnowTran, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.01 | — | |
| COLAModel=gemma-2-9b-it, Calibration Category=Math, Pruning Method=Wanda, Sparsity=25%2026.03 | 23.01 | — | |
| COLAModel=gemma-2-9b-it, Calibration Category=Mean, Pruning Method=Wanda, Sparsity=25%2026.03 | 22.51 | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=NLI, Pruning Method=Wanda, Sparsity=25%2026.03 | 22.02 | — | |
| COLAModel=gemma-2-9b-it, Calibration Category=CommQA, Pruning Method=Wanda, Sparsity=25%2026.03 | 22.02 | — | |
| COLAModel=Llama-3.1-8B-Instruct, Calibration Category=NLI, Pruning Method=Wanda, Sparsity=25%2026.03 | 21.85 | — | |
| COLAModel=gemma-2-9b-it, Calibration Category=KnowTran, Pruning Method=Wanda, Sparsity=25%2026.03 | 21.85 | — | |
| DenseModel=gemma-2-9b-it, Pruning Method=None, Sparsity=0%2026.03 | 21.48 | — | |
| ZipCalModel=Llama-3.1-8B-Instruct, Calibration Category=CommQA, Pruning Method=Wanda, Sparsity=25%2026.03 | 21.24 | — | |
| COLAModel=gemma-2-9b-it, Calibration Category=NLI, Pruning Method=Wanda, Sparsity=25%2026.03 | 21.24 | — |