Autonomous Machine Learning Engineering on MLE-Bench Lite
81.82Any Medal RateAiScientist
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| AiScientistModel=Gemini-3-Flash, Evaluation Context=Controlled Evaluation2026.04 | 81.82 | 100 | 86.36 | 18.18 | 31.82 | 31.82 | — | — | |
| AiScientistModel=GLM-5, Evaluation Context=Controlled Evaluation2026.04 | 81.82 | 100 | 90.91 | 9.09 | 31.82 | 40.91 | — | — | |
| AIBuildAIModel=Claude-Opus-4.6, Evaluation Context=Official Leaderboard Results2026.04 | 77.27 | 100 | 81.82 | 13.64 | 25.76 | 37.88 | — | — | |
| LoongFlowModel=Gemini-3-Flash, Evaluation Context=Controlled Evaluation2026.04 | 77.27 | 77.27 | 77.27 | 12.12 | 25.76 | 39.39 | — | — | |
| ML-Master 2.0Model=DeepSeekV3.2-Spe, Evaluation Context=Official Leaderboard Results2026.04 | 75.76 | 100 | 84.85 | 13.64 | 31.82 | 30.3 | — | — | |
| Famou-Agent 2.0Model=Gemini-2.5-Pro, Evaluation Context=Official Leaderboard Results2026.04 | 75.76 | 100 | 86.36 | 15.15 | 19.7 | 40.91 | — | — | |
| EvoMasterBackend model=GPT-5.4, Maximum runtime=24 hours2026.04 | 75.76 | 100 | 84.85 | 13.64 | 31.82 | 30.3 | — | — | |
| MARSModel=Gemini-3-Pro, Evaluation Context=Official Leaderboard Results2026.04 | 74.24 | 100 | 89.39 | 6.06 | 15.15 | 53.03 | — | — | |
| LeerooBase Model=Gemini-3-pro, Time Budget=24h, Open-world protocol=true2026.03 | 68.2 | — | — | — | — | — | — | — | |
| GOMEBase Model=GPT-5, Time Budget=12h, Open-world protocol=false2026.03 | 68.2 | — | — | — | — | — | — | — | |
| R&D-AgentModel=GPT-5, Evaluation Context=Official Leaderboard Results2026.04 | 68.18 | 77.27 | 74.24 | 12.12 | 22.73 | 33.33 | — | — | |
| LeerooModel=Gemini-3-Pro, Evaluation Context=Official Leaderboard Results2026.04 | 68.18 | 68.18 | 68.18 | 18.18 | 19.7 | 30.3 | — | — | |
| MLE-STAR2026.04 | 68.18 | 100 | 75.76 | 16.67 | 16.67 | 34.85 | — | — | |
| R&D-Agent2026.04 | 68.18 | 77.27 | 74.24 | 12.12 | 22.73 | 33.33 | — | — | |
| ThesisBase Model=GPT-5-Codex, Time Budget=24h, Open-world protocol=true2026.03 | 65.2 | — | — | — | — | — | — | — | |
| ML-Master 2.0Model=GLM-5, Evaluation Context=Controlled Evaluation2026.04 | 63.64 | 100 | 81.82 | 18.18 | 13.64 | 31.82 | — | — | |
| InternAgentModel=DeepSeek-R1, Evaluation Context=Official Leaderboard Results2026.04 | 62.12 | 100 | 78.79 | 10.61 | 16.67 | 34.85 | — | — | |
| InternAgent-MLEBase Model=DeepSeek-R1, Time Budget=12h, Open-world protocol=true2026.03 | 62.1 | — | — | — | — | — | — | — | |
| FM AgentBase Model=Gemini-2.5-pro, Time Budget=24h, Open-world protocol=true2026.03 | 62.1 | — | — | — | — | — | — | — | |
| CLAUDE-4.5-SONNET2026.02 | 60.61 | 100 | 80.3 | 7.58 | 16.67 | 36.36 | 75.86 | — | |
| GEMINI-3-PRO2026.02 | 59.09 | 100 | 84.85 | 4.55 | 15.15 | 39.39 | 79.18 | — | |
| GPT-5.22026.02 | 56.06 | 95.45 | 71.21 | 10.61 | 16.67 | 28.79 | 71.05 | — | |
| AIRABase Model=o3, Time Budget=24h, Open-world protocol=false2026.03 | 55 | — | — | — | — | — | — | — | |
| AIRA-dojoModel=o3, Evaluation Context=Official Leaderboard Results2026.04 | 55 | 100 | 70.45 | 7.95 | 12.73 | 34.32 | — | — | |
| ACE-30BOpen-Source=Best performance2026.02 | 51.52 | 100 | 78.79 | 12.12 | 7.58 | 31.82 | 71.14 | — | |
| KompeteAIBase Model=Gemini-2.5-flash, Time Budget=6h, Open-world protocol=true2026.03 | 51.5 | — | — | — | — | — | — | — | |
| ML-MasterBase Model=Deepseek-R1, Evaluation Protocol=pass@12026.03 | 48.5 | — | — | — | — | 18.1 | — | — | |
| ML-MasterModel=DeepSeek-R1, Evaluation Context=Official Leaderboard Results2026.04 | 48.48 | 100 | 74.24 | 4.55 | 13.64 | 30.3 | — | — | |
| AIRABase Model=o3-preview, Evaluation Protocol=pass@12026.03 | 47.73 | — | — | — | — | 28.64 | — | — | |
| + IdleSpec (Ours)Use idle?=true, Backbone=Gemini-2.5-Flash2026.05 | 45.5 | 95.5 | 59.1 | 13.6 | 13.6 | 18.2 | — | 86.4 | |
| SciDEREvaluation Protocol=pass@12026.03 | 45.45 | — | — | — | — | 36.4 | — | — | |
| AIDEModel=Gemini-3-Flash, Evaluation Context=Controlled Evaluation2026.04 | 45.45 | 77.27 | 54.55 | 4.55 | 9.09 | 31.82 | — | — | |
| AIDEModel=GLM-5, Evaluation Context=Controlled Evaluation2026.04 | 40.91 | 77.27 | 50 | 4.55 | 13.64 | 22.73 | — | — | |
| +Sequential RevisionUse idle?=false, Backbone=Gemini-2.5-Flash2026.05 | 40.9 | 90.1 | 50 | 13.6 | 9.1 | 18.2 | — | 81.8 | |
| DEEPSEEK-V3.22026.02 | 39.39 | 98.48 | 71.21 | 7.58 | 4.55 | 27.27 | 65.92 | — | |
| QWEN3-235B-2507full name=Qwen3-235B-A22B-Thinking-25072026.02 | 37.88 | 92.42 | 69.7 | 6.06 | 7.58 | 24.24 | 69.7 | — | |
| VanillaUse idle?=false, Backbone=Gemini-2.5-Flash2026.05 | 36.4 | 86.4 | 40.1 | 9.1 | 9.1 | 18.2 | — | 77.3 | |
| +Sleep-time ComputeUse idle?=true, Backbone=Gemini-2.5-Flash2026.05 | 36.4 | 90.1 | 59.1 | 13.6 | 9.1 | 13.6 | — | 81.8 | |
| Claude-4.5-SonnetFramework=ReAct2026.04 | 31.8 | 95.5 | 31.8 | 0 | 4.5 | 27.3 | — | — | |
| GLM-4.62026.02 | 30.3 | 90.91 | 45.45 | 6.06 | 3.03 | 21.21 | 45.45 | — | |
| Qwen3-14B-SFT-SandMLEModel Scale=14B, Framework=ReAct, Training Pipeline=SFT-SandMLE2026.04 | 27.3 | 95.5 | 31.8 | 4.5 | 0 | 22.7 | — | — | |
| Qwen3-30B-SandMLEModel Scale=30B, Framework=ReAct, Training Pipeline=SandMLE2026.04 | 27.3 | 100 | 36.4 | 0 | 13.6 | 13.6 | — | — | |
| Qwen3-30B-SFT-SandMLEModel Scale=30B, Framework=ReAct, Training Pipeline=SFT-SandMLE2026.04 | 27.3 | 90.9 | 45.5 | 4.5 | 9.1 | 13.6 | — | — | |
| QWEN3-30B-2507full name=Qwen3-30B-A3B-Thinking-25072026.02 | 27.27 | 84.85 | 50 | 1.52 | 7.58 | 18.18 | 48.75 | — | |
| Deepseek-V3.1Framework=ReAct2026.04 | 22.7 | 90.9 | 36.4 | 0 | 4.5 | 18.2 | — | — | |
| Gemini-2.5-flashFramework=ReAct2026.04 | 22.7 | 81.8 | 45.5 | 4.5 | 4.5 | 13.6 | — | — | |
| Qwen3-8B-SandMLEModel Scale=8B, Framework=ReAct, Training Pipeline=SandMLE2026.04 | 22.7 | 63.6 | 27.3 | 0 | 4.5 | 18.2 | — | — | |
| Qwen3-8B-SFT-SandMLEModel Scale=8B, Framework=ReAct, Training Pipeline=SFT-SandMLE2026.04 | 22.7 | 90.9 | 31.8 | 4.5 | 0 | 18.2 | — | — | |
| Qwen3-14B-SandMLEModel Scale=14B, Framework=ReAct, Training Pipeline=SandMLE2026.04 | 22.7 | 77.3 | 27.3 | 4.5 | 4.5 | 13.6 | — | — | |
| Qwen3-30B-Seed-SFTModel Scale=30B, Framework=ReAct, Training Pipeline=Seed-SFT2026.04 | 22.7 | 77.3 | 31.8 | 4.5 | 4.5 | 13.6 | — | — | |
| Qwen3-14B-BaseModel Scale=14B, Framework=ReAct, Training Pipeline=Base2026.04 | 18.2 | 72.7 | 18.2 | 4.5 | 4.5 | 18.2 | — | — | |
| Qwen3-14B-Seed-SFTModel Scale=14B, Framework=ReAct, Training Pipeline=Seed-SFT2026.04 | 18.2 | 72.7 | 22.7 | 4.5 | 0 | 13.6 | — | — | |
| OpenClawBackend model=GPT-5.4, Maximum runtime=24 hours2026.04 | 18.18 | 81.82 | 31.82 | 4.55 | 13.64 | 0 | — | — | |
| AIDEBase Model=o1-preview, Evaluation Protocol=pass@12026.03 | 16.9 | — | — | — | — | 9.4 | — | — | |
| Qwen3-8B-BaseModel Scale=8B, Framework=ReAct, Training Pipeline=Base2026.04 | 13.6 | 68.2 | 18.2 | 0 | 4.5 | 9.1 | — | — | |
| Qwen3-8B-Seed-SFTModel Scale=8B, Framework=ReAct, Training Pipeline=Seed-SFT2026.04 | 13.6 | 72.7 | 18.2 | 4.5 | 0 | 9.1 | — | — | |
| Qwen3-30B-BaseModel Scale=30B, Framework=ReAct, Training Pipeline=Base2026.04 | 13.6 | 68.2 | 18.2 | 4.5 | 0 | 9.1 | — | — |