Zero-shot Natural Language Understanding on ARC, BoolQ, HellaSwag, LAMBADA, PIQA, RACE, SciQ, Record, OBQA
46.8ARC ChallengeLlama 7B Baseline
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama 7B BaselineModel=Llama-3 7B, Method=Dense, Zero-shot=true2026.02 | 46.8 | 73.7 | — | 76.9 | — | 79.9 | — | — | — | 43 | 65.1 | 70 | |
| Llama 7B SparseModel=Llama-3 7B, Method=Venom Sparsity, Sparse steps=10k, Dense steps=38k, Zero-shot=true2026.02 | 46.4 | 73.8 | — | 76.4 | — | 79.3 | — | — | — | 46 | 65.2 | 69.1 | |
| Llama 1B SparseModel=Llama-3 1B, Method=Venom Sparsity, Sparse steps=30k, Dense steps=30k, Zero-shot=true2026.02 | 32.5 | 60.8 | — | 56.2 | — | 73 | — | — | — | 34.6 | 52.7 | 59 | |
| Llama 1B BaselineModel=Llama-3 1B, Method=Dense, Zero-shot=true2026.02 | 32.3 | 60.2 | — | 56.3 | — | 74.1 | — | — | — | 36.2 | 52.7 | 57.1 | |
| MaxScore-IModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 20.9 | 43.22 | 61.71 | 32.51 | 21.66 | 67.41 | 28.42 | 69.9 | 63.61 | 18.4 | 42.77 | — | |
| MaxScoreModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 20.73 | 44.49 | 62.23 | 32.85 | 23.27 | 67.41 | 28.52 | 72.5 | 64 | 18.4 | 43.44 | — | |
| DeepSeekModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 19.88 | 44.28 | 60.55 | 32.23 | 21.93 | 66.97 | 27.94 | 70.9 | 62.57 | 17.6 | 42.49 | — | |
| GShard-IModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 19.8 | 44.36 | 59.94 | 32.54 | 21.52 | 67.03 | 28.23 | 68.7 | 62.84 | 16 | 42.1 | — | |
| ExpertChoiceModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 19.37 | 42 | 61.74 | 32.1 | 21.19 | 66.16 | 27.18 | 68.4 | 62.26 | 17.6 | 41.8 | — | |
| DropLessModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 19.28 | 44.07 | 61.16 | 32.03 | 21.35 | 67.14 | 27.08 | 67.9 | 61.55 | 16 | 41.76 | — | |
| GShardModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 18.86 | 44.49 | 61.9 | 31.74 | 21.54 | 66.38 | 28.52 | 69.4 | 62.08 | 16.2 | 42.11 | — | |
| DenseModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 18.69 | 40.19 | 57.06 | 28.91 | 16.28 | 63.71 | 25.65 | 64.2 | 56.05 | 15 | 38.57 | — | |
| SBASEModel size=base-sized, Context length=512, Batch size=688, Training tokens=65B, Precision=bfloat16, Evaluation Protocol=Zero-shot2025.08 | 18.34 | 43.73 | 57.61 | 30.96 | 19.7 | 65.18 | 27.37 | 68.3 | 60.06 | 16.2 | 40.75 | — |