Language Modeling Downstream Evaluation on Downstream tasks Average (test)
58.02Average ScoreBlockPruner
Evaluation Results
| Method | Links | |
|---|---|---|
| BlockPrunerModel=Mistral-7B-v0.3, Ratio (%)=25.122024.06 | 58.02 | |
| ShortGPTModel=Mistral-7B-v0.3, Ratio (%)=24.072024.06 | 57.4 | |
| BlockPrunerModel=Mistral-7B-v0.3, Ratio (%)=27.552024.06 | 57.02 | |
| BlockPrunerModel=Mistral-7B-v0.3, Ratio (%)=29.982024.06 | 56.1 | |
| ShortGPTModel=Mistral-7B-v0.3, Ratio (%)=27.082024.06 | 53.26 | |
| BlockPrunerModel=LLaMA3-8B, Ratio (%)=24.862024.06 | 52.36 | |
| BlockPrunerModel=LLaMA3-8B, Ratio (%)=27.582024.06 | 50.08 | |
| BlockPrunerModel=Mistral-7B-v0.3, Ratio (%)=36.002024.06 | 49.18 | |
| g=4, 2× widthTotal Parameters=4.71B, g=4, width=2×, Saved=33.8%2026.06 | 48.2 | |
| g=4Total Parameters=3.50B, g=4, Saved=50.9%2026.06 | 47.5 | |
| baseline (g=1)Total Parameters=7.12B, g=12026.06 | 47.4 | |
| BlockPrunerModel=LLaMA3-8B, Ratio (%)=30.302024.06 | 46.38 | |
| BlockPrunerModel=LLaMA3-8B, Ratio (%)=35.732024.06 | 45.41 | |
| ShortGPTModel=LLaMA3-8B, Ratio (%)=27.162024.06 | 44.51 | |
| OLMoE-1B-7B-0924Total Parameters=6.9B, training_tokens=≈20B2026.06 | 44.4 | |
| ShortGPTModel=LLaMA3-8B, Ratio (%)=29.882024.06 | 43.71 | |
| ShortGPTModel=LLaMA3-8B, Ratio (%)=35.312024.06 | 42.85 | |
| ShortGPTModel=Mistral-7B-v0.3, Ratio (%)=30.092024.06 | 42.79 | |
| ShortGPTModel=LLaMA3-8B, Ratio (%)=24.452024.06 | 41.97 | |
| ShortGPTModel=Mistral-7B-v0.3, Ratio (%)=36.112024.06 | 41.41 | |
| Relative Magnitude (RM)Model=LLaMA3-8B, Ratio (%)=24.452024.06 | 39.34 | |
| Relative Magnitude (RM)Model=LLaMA3-8B, Ratio (%)=27.162024.06 | 38.9 | |
| Relative Magnitude (RM)Model=Mistral-7B-v0.3, Ratio (%)=24.072024.06 | 37.75 | |
| Relative Magnitude (RM)Model=LLaMA3-8B, Ratio (%)=29.882024.06 | 37.48 | |
| Relative Magnitude (RM)Model=LLaMA3-8B, Ratio (%)=35.312024.06 | 36.8 | |
| Relative Magnitude (RM)Model=Mistral-7B-v0.3, Ratio (%)=30.092024.06 | 36.49 | |
| Relative Magnitude (RM)Model=Mistral-7B-v0.3, Ratio (%)=27.082024.06 | 35.89 | |
| Relative Magnitude (RM)Model=Mistral-7B-v0.3, Ratio (%)=36.112024.06 | 35.52 |