Reasoning on Reasoning Suite (OpenbookQA, ARC, WinoGrande, HellaSwag, PIQA, MathQA)
34OpenBookQA AccuracyDense
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DenseBackbone=LLaMA3.1-8B-Instruct, Compression Ratio=0.0%, Zero-shot=true2024.12 | 34 | 82 | 74 | 59 | 52 | 80 | 39 | 60 | |
| DenseBackbone=Mistral-7B-v0.1, Compression Ratio=0.0%, Zero-shot=true2024.12 | 33 | 81 | 74 | 61 | 50 | 81 | 36 | 59 | |
| DenseBackbone=LLaMA2-7B, Compression Ratio=0.0%, Zero-shot=true2024.12 | 32 | 69 | 67 | 57 | 40 | 78 | 28 | 53 | |
| DenseBackbone=LLaMA-2-13B, Compression Ratio=0.0%, Zero-shot=true2024.12 | 32 | 73 | 70 | 60 | 46 | 79 | 30 | 56 | |
| SliceGPTBackbone=LLaMA-2-13B, Compression Ratio=20.0%, Zero-shot=true2024.12 | 29 | 59 | 65 | 39 | 32 | 64 | 24 | 45 | |
| LaCoBackbone=LLaMA-2-13B, Compression Ratio=19.5%, Zero-shot=true2024.12 | 28 | 52 | 63 | 43 | 33 | 70 | 25 | 45 | |
| LaCoBackbone=LLaMA2-7B, Compression Ratio=18.1%, Zero-shot=true2024.12 | 26 | 48 | 59 | 42 | 32 | 69 | 24 | 43 | |
| GRASPBackbone=LLaMA-2-13B, Compression Ratio=20.0%, Zero-shot=true2024.12 | 26 | 61 | 66 | 47 | 35 | 73 | 24 | 47 | |
| LaCoBackbone=LLaMA3.1-8B-Instruct, Compression Ratio=19.0%, Zero-shot=true2024.12 | 26 | 49 | 65 | 33 | 30 | 65 | 30 | 42 | |
| GRASPBackbone=LLaMA2-7B, Compression Ratio=21.6%, Zero-shot=true2024.12 | 24 | 54 | 63 | 43 | 33 | 71 | 23 | 44 | |
| ShortGPTBackbone=LLaMA-2-13B, Compression Ratio=22.1%, Zero-shot=true2024.12 | 24 | 50 | 63 | 46 | 33 | 70 | 24 | 44 | |
| ShortGPTBackbone=LLaMA2-7B, Compression Ratio=21.1%, Zero-shot=true2024.12 | 23 | 49 | 63 | 42 | 31 | 68 | 23 | 43 | |
| SliceGPTBackbone=LLaMA2-7B, Compression Ratio=21.5%, Zero-shot=true2024.12 | 22 | 54 | 61 | 37 | 28 | 63 | 23 | 41 | |
| GRASPBackbone=LLaMA3.1-8B-Instruct, Compression Ratio=20.0%, Zero-shot=true2024.12 | 22 | 60 | 70 | 44 | 37 | 69 | 28 | 47 | |
| GRASPBackbone=Mistral-7B-v0.1, Compression Ratio=20.0%, Zero-shot=true2024.12 | 21 | 56 | 68 | 43 | 38 | 67 | 26 | 46 | |
| ShortGPTBackbone=LLaMA3.1-8B-Instruct, Compression Ratio=21.7%, Zero-shot=true2024.12 | 21 | 57 | 66 | 42 | 32 | 67 | 26 | 44 | |
| LaCoBackbone=Mistral-7B-v0.1, Compression Ratio=21.1%, Zero-shot=true2024.12 | 20 | 35 | 58 | 26 | 25 | 53 | 24 | 34 | |
| ShortGPTBackbone=Mistral-7B-v0.1, Compression Ratio=21.1%, Zero-shot=true2024.12 | 19 | 57 | 68 | 46 | 37 | 71 | 26 | 46 | |
| SliceGPTBackbone=Mistral-7B-v0.1, Compression Ratio=20.0%, Zero-shot=true2024.12 | 19 | 51 | 59 | 35 | 25 | 61 | 23 | 39 | |
| SliceGPTBackbone=LLaMA3.1-8B-Instruct, Compression Ratio=20.0%, Zero-shot=true2024.12 | 15 | 43 | 51 | 30 | 23 | 58 | 22 | 35 |