Question Answering on SciQ (Acc Norm, Acc)
86.4Acc NormSoftmax
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SoftmaxQuantization=8-bit2025.04 | 86.4 | 90 | |
| SoftmaxModel Scale=1.8B2025.04 | 86.4 | 90.1 | |
| SoftmaxQuantization=2-bit2025.04 | 86.3 | 90.2 | |
| SoftmaxQuantization=3-bit2025.04 | 86.3 | 90.2 | |
| SoftmaxQuantization=4-bit2025.04 | 85.9 | 90.5 | |
| SoftmaxQuantization=3-bit2025.04 | 82.2 | 88 | |
| SoftpickQuantization=4-bit2025.04 | 80.4 | 86.6 | |
| SoftpickQuantization=8-bit2025.04 | 80.4 | 87.2 | |
| SoftpickModel Scale=1.8B2025.04 | 80.4 | 87 | |
| SoftpickQuantization=2-bit2025.04 | 80.4 | 87.1 | |
| SoftpickQuantization=3-bit2025.04 | 80.4 | 87.1 | |
| SoftpickQuantization=3-bit2025.04 | 78.3 | 85.5 | |
| SoftpickQuantization=8-bit2025.04 | 77.6 | 83.7 | |
| SoftpickQuantization=8-bit2025.04 | 77.4 | 83.5 | |
| SoftpickModel Scale=340M2025.04 | 77.3 | 83.6 | |
| SoftpickQuantization=4-bit2025.04 | 76.4 | 82.7 | |
| SoftpickQuantization=3-bit2025.04 | 75.9 | 82.3 | |
| SoftmaxQuantization=8-bit2025.04 | 75.2 | 83.4 | |
| SoftpickQuantization=4-bit2025.04 | 75.1 | 82.8 | |
| SoftmaxQuantization=8-bit2025.04 | 75 | 83.2 | |
| SoftmaxQuantization=4-bit2025.04 | 74.9 | 82.9 | |
| SoftmaxModel Scale=340M2025.04 | 74.9 | 83.2 | |
| SoftmaxQuantization=3-bit2025.04 | 73.5 | 79.3 | |
| SoftmaxQuantization=4-bit2025.04 | 73.4 | 81 | |
| SoftpickQuantization=3-bit2025.04 | 72.1 | 80.7 | |
| SoftmaxQuantization=3-bit2025.04 | 71.6 | 80.5 | |
| SoftpickQuantization=2-bit2025.04 | 51.4 | 53.6 | |
| SoftmaxQuantization=2-bit2025.04 | 45.8 | 43.9 | |
| SoftpickQuantization=2-bit2025.04 | 45.3 | 43.8 | |
| SoftpickQuantization=2-bit2025.04 | 35.8 | 34.8 | |
| SoftmaxQuantization=2-bit2025.04 | 29.8 | 27.8 | |
| SoftmaxQuantization=2-bit2025.04 | 22.2 | 21.6 |