Zero-shot Evaluation on ARC-E, BoolQ, HSwag, LAMBADA, OBQA, PIQA, SocIQA, and WinoGr (test)
60.65ARC-E AccuracyXSA
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| XSAModel size=2.7B, Protocol=Zero-shot2026.03 | 60.65 | 64.86 | 67.4 | 62.04 | 38.4 | 77.8 | 41.45 | 62.75 | 59.42 | 1.36 | |
| XSAModel size=1.3B, Protocol=Zero-shot2026.03 | 58.84 | 62.29 | 62.41 | 58.57 | 36 | 76.61 | 42.84 | 59.98 | 57.19 | 1.03 | |
| Transformer BaselineModel size=2.7B, Protocol=Zero-shot2026.03 | 58.59 | 60.98 | 66.2 | 60.18 | 37 | 76.61 | 42.94 | 61.96 | 58.06 | — | |
| Transformer BaselineModel size=1.3B, Protocol=Zero-shot2026.03 | 56.19 | 65.47 | 60.69 | 56.24 | 34.6 | 75.9 | 41.4 | 58.8 | 56.16 | — | |
| XSAModel size=0.7B, Protocol=Zero-shot2026.03 | 52.69 | 61.19 | 56.29 | 54.07 | 32.2 | 73.78 | 41.45 | 56.2 | 53.48 | 0.26 | |
| Transformer BaselineModel size=0.7B, Protocol=Zero-shot2026.03 | 51.26 | 61.07 | 55.68 | 52.82 | 35 | 74.05 | 40.02 | 55.88 | 53.22 | — |