Reading Comprehension on DROP (DROP metric)
48.68DROP ScoreMONA
Evaluation Results
| Method | Links | |
|---|---|---|
| MONAModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=Chain of Thought (CoT)2026.05 | 48.68 | |
| MuonModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=Chain of Thought (CoT)2026.05 | 47.57 | |
| AdamWModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=Chain of Thought (CoT)2026.05 | 45.6 | |
| Warmup-Stable-Only (WSO)Model=8B, αpre=1.0, αmid=1.02026.03 | 36.4 | |
| WSDModel=8B, αpre=1.0, αmid=0.02026.03 | 32.6 | |
| LinearModel=8B, αpre=0.1, αmid=1.02026.03 | 30.5 | |
| WSDModel=8B, αpre=0.1, αmid=1.02026.03 | 29.9 | |
| CosineModel=8B, αpre=0.1, αmid=1.02026.03 | 28.8 | |
| WSDModel=8B, αpre=0.1, αmid=0.02026.03 | 28.3 | |
| LinearModel=8B, αpre=0.1, αmid=0.02026.03 | 28.3 | |
| CosineModel=8B, αpre=0.1, αmid=0.02026.03 | 26.9 | |
| HYPERCONNECTIONSInput length=2k tokens, Evaluation protocol=Arora et al. [2024]2026.05 | 24.86 | |
| WSDModel=1B, αpre=0.1, αmid=1.02026.03 | 23.7 | |
| RESFORMERInput length=2k tokens, Evaluation protocol=Arora et al. [2024]2026.05 | 22.8 | |
| Warmup-Stable-Only (WSO)Model=1B, αpre=1.0, αmid=1.02026.03 | 22.7 | |
| DENSEFORMERInput length=2k tokens, Evaluation protocol=Arora et al. [2024]2026.05 | 22.66 | |
| WSDModel=1B, αpre=1.0, αmid=0.02026.03 | 22 | |
| WSDModel=1B, αpre=0.1, αmid=0.02026.03 | 22 | |
| SATFORMERInput length=2k tokens, Evaluation protocol=Arora et al. [2024]2026.05 | 21.41 | |
| MUDDFORMERInput length=2k tokens, Evaluation protocol=Arora et al. [2024]2026.05 | 20.84 | |
| TRANSFORMERInput length=2k tokens, Evaluation protocol=Arora et al. [2024]2026.05 | 20.07 | |
| CosineModel=1B, αpre=0.1, αmid=0.02026.03 | 17.6 | |
| CosineModel=1B, αpre=0.1, αmid=1.02026.03 | 17.1 | |
| LinearModel=1B, αpre=0.1, αmid=1.02026.03 | 16.3 | |
| LinearModel=1B, αpre=0.1, αmid=0.02026.03 | 16.2 |