Zero-shot Evaluation on ARC-e, BoolQ, HellaSwag, LAMBADA, PIQA, RACE, SocialIQA, SciQ, and SWAG
77.9ARC-e AccuracyJREG
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| JREGSize=3.4B, alpha=1.02026.01 | 77.9 | 61.6 | 50.3 | 57.2 | 75.2 | 37.8 | 42.5 | 93 | 51.5 | 60.8 | |
| BaselineSize=3.4B, alpha=N/A2026.01 | 75.6 | 59.4 | 49.7 | 55.9 | 76.5 | 36.3 | 43.9 | 93.4 | 51.3 | 60.2 | |
| JREGSize=1B, alpha=3.02026.01 | 70.6 | 59.3 | 42.6 | 45.6 | 71.8 | 34.9 | 41.4 | 91.2 | 47.3 | 56.1 | |
| JREGSize=1B, alpha=1.02026.01 | 69.4 | 61.4 | 43.1 | 47.2 | 71.8 | 35.3 | 42.1 | 88.6 | 47.6 | 56.2 | |
| BaselineSize=1B, alpha=N/A2026.01 | 68.5 | 61.4 | 42.9 | 46.6 | 72.4 | 32.9 | 41 | 89.5 | 47.3 | 55.8 | |
| JREGSize=170M, alpha=1.02026.01 | 57.2 | 60 | 32.1 | 31.7 | 65.2 | 29.9 | 38.8 | 81.1 | 40.2 | 48.5 | |
| JREGSize=170M, alpha=3.02026.01 | 57.2 | 60 | 32.4 | 31.6 | 65.1 | 28.9 | 40 | 81.7 | 39.9 | 48.5 | |
| JREGSize=170M, alpha=0.52026.01 | 57 | 57.1 | 32.5 | 30.7 | 65 | 29.3 | 38.6 | 82.2 | 40 | 48 | |
| JREGSize=170M, alpha=0.12026.01 | 56.1 | 60.5 | 32.2 | 29.8 | 65.6 | 29.6 | 39.2 | 80.9 | 39.9 | 48.2 | |
| JREGSize=170M, alpha=0.32026.01 | 56 | 59.2 | 32.2 | 31.7 | 65 | 29.1 | 38.1 | 82.2 | 40 | 48 | |
| JREGSize=170M, alpha=0.02026.01 | 55.7 | 58.3 | 32.4 | 32.5 | 65.9 | 29.6 | 38.6 | 81.3 | 40.2 | 48.3 | |
| BaselineSize=170M, alpha=N/A2026.01 | 54.9 | 57.5 | 32.1 | 32 | 64 | 29.1 | 38.4 | 80.2 | 39.7 | 47.5 |