Reasoning on GPQA (Accuracy)
63.6AccuracyMOCHA
Evaluation Results
| Method | Links | |
|---|---|---|
| MOCHAEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 63.6 | |
| Seed SkillEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 59.2 | |
| TextGradEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 59.2 | |
| ProTeGiEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 59.2 | |
| GEPAEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 59.2 | |
| top-pModel=LLaMA3.1-8B-Instruct, Temperature=1.0, Few-shot=8-shot2025.09 | 32.81 | |
| top-HModel=Phi-3-Mini, Temperature=1.0, Few-shot=8-shot2025.09 | 32.37 | |
| min-pModel=Phi-3-Mini, Temperature=1.0, Few-shot=8-shot2025.09 | 31.92 | |
| top-HModel=Phi-3-Mini, Temperature=1.5, Few-shot=8-shot2025.09 | 30.8 | |
| top-HModel=Phi-3-Mini, Temperature=2.0, Few-shot=8-shot2025.09 | 30.8 | |
| top-HModel=LLaMA3.1-8B-Instruct, Temperature=1.5, Few-shot=8-shot2025.09 | 30.58 | |
| top-pModel=Phi-3-Mini, Temperature=1.0, Few-shot=8-shot2025.09 | 30.58 | |
| min-pModel=Qwen2.5 3B, Temperature=1.5, Few-shot=8-shot2025.09 | 30.13 | |
| min-pModel=Phi-3-Mini, Temperature=1.5, Few-shot=8-shot2025.09 | 29.91 | |
| top-HModel=LLaMA3.1-8B-Instruct, Temperature=1.0, Few-shot=8-shot2025.09 | 29.24 | |
| top-HModel=Qwen2.5 3B, Temperature=1.0, Few-shot=8-shot2025.09 | 28.79 | |
| top-HModel=LLaMA3.1-8B-Instruct, Temperature=2.0, Few-shot=8-shot2025.09 | 28.79 | |
| top-pModel=LLaMA3.1-8B-Instruct, Temperature=1.5, Few-shot=8-shot2025.09 | 28.57 | |
| top-pModel=Phi-3-Mini, Temperature=1.5, Few-shot=8-shot2025.09 | 28.57 | |
| min-pModel=Qwen2.5 3B, Temperature=1.0, Few-shot=8-shot2025.09 | 28.35 | |
| min-pModel=LLaMA3.1-8B-Instruct, Temperature=1.5, Few-shot=8-shot2025.09 | 28.35 | |
| top-HModel=Qwen2.5 3B, Temperature=2.0, Few-shot=8-shot2025.09 | 28.12 | |
| top-HModel=Qwen2.5 3B, Temperature=1.5, Few-shot=8-shot2025.09 | 27.9 | |
| top-pModel=Qwen2.5 3B, Temperature=1.0, Few-shot=8-shot2025.09 | 27.68 | |
| top-pModel=Qwen2.5 3B, Temperature=1.5, Few-shot=8-shot2025.09 | 27.23 | |
| min-pModel=LLaMA3.1-8B-Instruct, Temperature=1.0, Few-shot=8-shot2025.09 | 26.34 | |
| min-pModel=LLaMA3.1-8B-Instruct, Temperature=2.0, Few-shot=8-shot2025.09 | 26.12 | |
| min-pModel=Qwen2.5 3B, Temperature=2.0, Few-shot=8-shot2025.09 | 25 | |
| top-pModel=LLaMA3.1-8B-Instruct, Temperature=2.0, Few-shot=8-shot2025.09 | 23.88 | |
| min-pModel=Phi-3-Mini, Temperature=2.0, Few-shot=8-shot2025.09 | 23.44 | |
| top-pModel=Qwen2.5 3B, Temperature=2.0, Few-shot=8-shot2025.09 | 22.32 | |
| Llama3.2-3B-RLVRBackbone=Llama3.2-3B, Training Strategy=RLVR2026.05 | 22.2 | |
| top-pModel=Phi-3-Mini, Temperature=2.0, Few-shot=8-shot2025.09 | 18.53 | |
| Llama3.2-3B-GRLOBackbone=Llama3.2-3B, Training Strategy=GRLO2026.05 | 17.7 | |
| Llama3.2-3B-GRLO+RLVRBackbone=Llama3.2-3B, Training Strategy=GRLO+RLVR2026.05 | 17.7 | |
| Llama3.2-3B-InstructBackbone=Llama3.2-3B, Training Strategy=Instruct2026.05 | 10.1 | |
| Llama3.2-3B-SFTBackbone=Llama3.2-3B, Training Strategy=SFT2026.05 | 5.1 |