Science Question Answering on ARC Easy (Character-level Accuracy)
87.8Accuracy (Character-level)No pruning
Evaluation results
| Method | Links | |
|---|---|---|
| No pruningSparsity=0, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 87.8 | |
| Magnitude-DimSparsity=10%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 80.2 | |
| DIETSparsity=10%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 79.1 | |
| LFM2 1.2BParameter Count=1.2B, Evaluation Protocol=0-shot2026.03 | 77.9 | |
| Nemotron-Flash 1BParameter Count=1B, Evaluation Protocol=0-shot2026.03 | 74.83 | |
| LFM2 700MParameter Count=700M, Evaluation Protocol=0-shot2026.03 | 74.6 | |
| Gemma3 1BParameter Count=1B, Evaluation Protocol=0-shot2026.03 | 73 | |
| MobileLLM-Flash 1.4BParameter Count=1.4B, Evaluation Protocol=0-shot2026.03 | 72.26 | |
| MobileLLM-Flash 650MParameter Count=650M, Evaluation Protocol=0-shot2026.03 | 66.55 | |
| LFM2 350MParameter Count=350M, Evaluation Protocol=0-shot2026.03 | 66.04 | |
| Llama3.2 1BParameter Count=1B, Evaluation Protocol=0-shot2026.03 | 63.47 | |
| MobileLLM-Flash 350MParameter Count=350M, Evaluation Protocol=0-shot2026.03 | 63.26 | |
| Magnitude-DimSparsity=20%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 62 | |
| DIETSparsity=20%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 61.7 | |
| Qwen3 0.6BParameter Count=0.6B, Evaluation Protocol=0-shot2026.03 | 58.08 | |
| Gemma3 270MParameter Count=270M, Evaluation Protocol=0-shot2026.03 | 57.32 | |
| PuDDingSparsity=20%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 51.3 | |
| PuDDingSparsity=10%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 49.1 | |
| SliceGPTSparsity=20%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 27.4 | |
| SliceGPTSparsity=10%, Backbone=Gemma-2 9B, Evaluation Protocol=Zero-shot2026.03 | 27.2 |