Zero-Shot Natural Language Understanding on CSQA, SIQA, HS, WG, PIQA, OBQA, ARC-E, and ARC-C Suite
49.8CSQA AccuracySTAR-LDM
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| STAR-LDMParams=.96B, Scoring=LM + Diff Loss2026.02 | 49.8 | 46.8 | 39 | 51.4 | 66.1 | 41.3 | 55 | 31.4 | 47.6 | |
| STAR-LDMParams=.96B, Scoring=Diffusion Loss2026.02 | 48.7 | 46.8 | 37.9 | 50.4 | 65.5 | 39.3 | 55 | 32.1 | 47 | |
| GPT2-LargeParams=.77B, Scoring=LM Loss2026.02 | 36.6 | 42.1 | 42.9 | 51.9 | 69.2 | 34.2 | 46.6 | 25.1 | 43.6 | |
| GPT2-XLParams=1.5B, Scoring=LM Loss2026.02 | 36.3 | 42.3 | 47.9 | 53.1 | 70.5 | 34.4 | 51.1 | 28.5 | 45.5 | |
| Pytha-1.4bParams=1.4B, Scoring=LM Loss2026.02 | 35.7 | 43.2 | 50.9 | 54.2 | 71.1 | 36 | 54 | 28.3 | 46.7 | |
| Pytha-1bParams=1B, Scoring=LM Loss2026.02 | 35.3 | 42.4 | 45.6 | 52.6 | 69.2 | 32.4 | 49.1 | 27 | 44.2 | |
| STAR-LDMParams=.96B, Scoring=LM Loss2026.02 | 35.2 | 42.2 | 44 | 50.5 | 69.3 | 27.1 | 37.6 | 26.3 | 41.5 | |
| Random Baseline2026.02 | 20 | 33.3 | 25 | 50 | 50 | 25 | 25 | 25 | 31.7 |