Question Answering on ARC Easy (Accuracy and Normalized Accuracy)
67.26Normalized AccuracySoftmax
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SoftmaxQuantization=8-bit2025.04 | 67.26 | 72.22 | |
| SoftmaxQuantization=4-bit2025.04 | 66.75 | 71.93 | |
| SoftpickQuantization=8-bit2025.04 | 62.21 | 69.07 | |
| SoftpickQuantization=4-bit2025.04 | 58.84 | 67.05 | |
| SoftpickQuantization=8-bit2025.04 | 56.36 | 60.94 | |
| SoftmaxQuantization=8-bit2025.04 | 56.31 | 60.31 | |
| SoftpickQuantization=4-bit2025.04 | 54.29 | 59.26 | |
| SoftmaxQuantization=4-bit2025.04 | 53.96 | 59.93 | |
| SSAArchitecture=Nemotron-style decoder, Parameters=114M, Training steps=22K, Training dataset=FineWeb, Zero-shot=true2025.08 | 53.87 | — | |
| SoftpickQuantization=3-bit2025.04 | 52.57 | 57.95 | |
| SoftpickQuantization=3-bit2025.04 | 51.81 | 56.65 | |
| SoftmaxQuantization=3-bit2025.04 | 48.65 | 52.78 | |
| SoftmaxQuantization=3-bit2025.04 | 48.57 | 53.83 | |
| SoftpickQuantization=2-bit2025.04 | 36.36 | 37.54 | |
| Baselineshot=0-shot, eval-framework=lm-evaluation-harness2026.04 | 35.69 | 38.93 | |
| GMT v7shot=0-shot, eval-framework=lm-evaluation-harness2026.04 | 34.76 | 37.46 | |
| SoftpickQuantization=2-bit2025.04 | 29.76 | 29.88 | |
| SoftmaxArchitecture=Nemotron-style decoder, Parameters=114M, Training steps=22K, Training dataset=FineWeb, Zero-shot=true2025.08 | 29.34 | — | |
| SoftmaxQuantization=2-bit2025.04 | 27.27 | 27.02 | |
| SoftmaxQuantization=2-bit2025.04 | 26.68 | 26.81 |