Reasoning on GPQA Diamond (Accuracy)
58.08AccuracySequential MAS + MASPO
Evaluation Results
| Method | Links | |
|---|---|---|
| Sequential MAS + MASPOPrompt Opt.=true, Joint Opt.=true2026.05 | 58.08 | |
| Hierarchical MAS + MASPOPrompt Opt.=true, Joint Opt.=true2026.05 | 54.04 | |
| Hierarchical MAS + SPOPrompt Opt.=true, Joint Opt.=false2026.05 | 51.01 | |
| Hierarchical MASPrompt Opt.=false, Joint Opt.=false2026.05 | 50.63 | |
| Sequential MAS + SPOPrompt Opt.=true, Joint Opt.=false2026.05 | 49.52 | |
| Hierarchical MAS + TPEPrompt Opt.=true, Joint Opt.=false2026.05 | 49.49 | |
| Sequential MAS + TPEPrompt Opt.=true, Joint Opt.=false2026.05 | 48.04 | |
| AgentDropoutPrompt Opt.=false, Joint Opt.=false2026.05 | 47.98 | |
| Self-RefinePrompt Opt.=false, Joint Opt.=false2026.05 | 47.73 | |
| Sequential MASPrompt Opt.=false, Joint Opt.=false2026.05 | 47.73 | |
| SC (CoT)Prompt Opt.=false, Joint Opt.=false2026.05 | 47.52 | |
| CoTPrompt Opt.=false, Joint Opt.=false2026.05 | 46.72 | |
| VanillaPrompt Opt.=false, Joint Opt.=false2026.05 | 45.96 | |
| LRSModel=OPEN-REASONER-7B, Prompting=0-shot2026.05 | 39.4 | |
| few-shotModel=OPEN-REASONER-7B2026.05 | 38.4 | |
| BaseModel=Qwen-2.5-7B-Inst, Training Strategy=Base, Evaluation Protocol=0-shot2026.04 | 36.36 | |
| CoTModel=OPEN-REASONER-7B2026.05 | 35.9 | |
| POPModel=Qwen-2.5-7B-Inst, Training Strategy=POP, Evaluation Protocol=0-shot2026.04 | 34.85 | |
| BaseModel=Qwen-2.5-7B, Training Strategy=Base, Evaluation Protocol=0-shot2026.04 | 33.84 | |
| Train on DModel=Qwen-2.5-7B-Inst, Training Strategy=Naive pretraining on D, Evaluation Protocol=0-shot2026.04 | 32.83 | |
| Rubric-grounded GRPOCheckpoint=Best by held-out rubric reward, Backbone=Llama-3.1-8B-Instruct2026.05 | 32.32 | |
| BaseModel=OPEN-REASONER-7B, Prompting=0-shot2026.05 | 32.3 | |
| Train on DModel=Qwen-2.5-7B, Training Strategy=Naive pretraining on D, Evaluation Protocol=0-shot2026.04 | 31.82 | |
| POPModel=Qwen-2.5-7B, Training Strategy=POP, Evaluation Protocol=0-shot2026.04 | 30.81 | |
| LRS BASICModel=OPEN-REASONER-7B, Prompting=0-shot2026.05 | 30.8 | |
| Nautile-370MTraining tokens=∼0.8T, Evaluation Protocol=0-shot2026.04 | 27.3 | |
| Granite 350MTraining tokens=10–12T, Evaluation Protocol=0-shot2026.04 | 26.3 | |
| LFM2.5 350MTraining tokens=28T, Evaluation Protocol=0-shot2026.04 | 24.8 | |
| Llama-3.1-8B-InstructCheckpoint=Base2026.05 | 24.24 | |
| SmolLM2 360MTraining tokens=4T, Evaluation Protocol=0-shot2026.04 | 23.2 | |
| LRSModel=OPEN-REASONER-1.5B, Prompting=0-shot2026.05 | 22.8 | |
| BaseModel=OPEN-REASONER-1.5B, Prompting=0-shot2026.05 | 18.2 | |
| CoTModel=OPEN-REASONER-1.5B2026.05 | 17.2 | |
| few-shotModel=OPEN-REASONER-1.5B2026.05 | 17.2 | |
| LRS BASICModel=OPEN-REASONER-1.5B, Prompting=0-shot2026.05 | 15.7 | |
| Qwen2.5 0.5BTraining tokens=18T, Evaluation Protocol=0-shot2026.04 | 10.1 |