Autonomous Machine Learning Engineering on MLE-bench (held-in and held-out)
76.53CIFAR-10 PerformanceAIDE (GPT-5)
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AIDE (GPT-5)Scaffold=AIDE, Model=GPT-5, #Params=N/A, Paradigm=Prompt-Based2025.05 | 76.53 | 22.15 | 8.77 | 77.38 | 31.5 | 9.18 | 26.42 | 4.51 | 0.13 | 4.25 | 0.14 | 29.69 | 0.11 | 20.95 | |
| ML-Master (DeepSeek-R1)Scaffold=ML-Master, Model=DeepSeek-R1, #Params=671B, Paradigm=Prompt-Based2025.05 | 73.43 | 18.25 | 12.07 | 14.56 | 14.75 | 2.78 | 33.39 | 3.03 | 0 | 4.01 | 0.04 | 29.27 | 0.22 | 13.13 | |
| AIDE (DeepSeek-R1)Scaffold=AIDE, Model=DeepSeek-R1, #Params=671B, Paradigm=Prompt-Based2025.05 | 72.55 | 5.35 | 13.07 | 33.23 | 10.25 | 4.54 | 30.77 | 1.38 | 0.31 | 0.36 | 0.01 | 5.78 | 0.14 | 11.4 | |
| ML-Master (GPT-5)Scaffold=ML-Master, Model=GPT-5, #Params=N/A, Paradigm=Prompt-Based2025.05 | 71.64 | 22.3 | 10.54 | 10.96 | 23.88 | 2.48 | 67.07 | 6.38 | 0.79 | 10.41 | 0.35 | 26.49 | 0.25 | 19.12 | |
| MLAB (GPT-5)Scaffold=MLAB, Model=GPT-5, #Params=N/A, Paradigm=Prompt-Based2025.05 | 61.46 | 12.15 | 12.74 | 66 | 45.63 | 6.43 | 89.59 | 4.36 | 11.2 | 6.79 | 0 | 23.38 | 0.23 | 18.14 | |
| MLAB (Qwen3-235B)Scaffold=MLAB, Model=Qwen3-235B, #Params=235B, Paradigm=Prompt-Based2025.05 | 57.61 | 3.01 | 6.7 | 62.6 | 2.12 | 16.36 | 26.68 | 0.3 | 1.02 | 0.8 | 0.01 | 1.96 | 0.07 | 10.93 | |
| AIDE (Gemini-2.5-Pro)Scaffold=AIDE, Model=Gemini-2.5-Pro, #Params=N/A, Paradigm=Prompt-Based2025.05 | 53.59 | 11.13 | 9.44 | 62.72 | 84.25 | 6.08 | 56.45 | 7.35 | 0.74 | 4.34 | 0.04 | 31.92 | 0.13 | 11.35 | |
| ML-AgentScaffold=ML-Agent (Ours), Model=7B, #Params=7B, Paradigm=Learning-Based2025.05 | 33.8 | 6.77 | 13.47 | 52.38 | 13.87 | 1.41 | 72.89 | 1.91 | 1.74 | 1.76 | 0.01 | 12.96 | 0.2 | 16.4 | |
| MLAB (DeepSeek-R1)Scaffold=MLAB, Model=DeepSeek-R1, #Params=671B, Paradigm=Prompt-Based2025.05 | 28.96 | 3.45 | 5.53 | 8.83 | 4.85 | 0.04 | 33.44 | 0.05 | 0.25 | 0.89 | 0 | 2.67 | 0.13 | 6.83 | |
| MLAB (Gemini-2.5-Pro)Scaffold=MLAB, Model=Gemini-2.5-Pro, #Params=N/A, Paradigm=Prompt-Based2025.05 | 16.78 | 1.16 | 0.1 | 37.85 | 4.38 | 4.26 | 22.38 | 0 | 0.13 | 0.04 | 0 | 0.13 | 0 | 5.38 | |
| AIDE (Qwen2.5-7B-Instruct)Scaffold=AIDE, Model=Qwen2.5-7B-Instruct, #Params=7B, Paradigm=Prompt-Based2025.05 | 11.36 | 2.42 | 7.52 | 7.33 | 4.75 | 4.33 | 0.52 | 9.78 | 0.38 | 0.07 | 0.01 | 0 | 0.08 | 0.77 | |
| MLAB (Qwen2.5-7B-Instruct)Scaffold=MLAB, Model=Qwen2.5-7B-Instruct, #Params=7B, Paradigm=Prompt-Based2025.05 | 1.37 | 0.23 | 1.39 | 2.1 | 2.52 | 6.32 | 12.25 | 1.23 | 0.51 | 0.46 | 0.06 | 3.75 | 0.04 | 1.43 | |
| ML-Master (Qwen2.5-7B-Instruct)Scaffold=ML-Master, Model=Qwen2.5-7B-Instruct, #Params=7B, Paradigm=Prompt-Based2025.05 | 1.03 | 0 | 0.1 | 2.44 | 1.38 | 3.99 | 1.12 | 1.79 | 0.26 | 0.04 | 0.02 | 0.02 | 0 | 0.1 | |
| AIDE (Qwen3-235B)Scaffold=AIDE, Model=Qwen3-235B, #Params=235B, Paradigm=Prompt-Based2025.05 | 0.1 | 2.04 | 11.1 | 41.65 | 4.75 | 2.89 | 8.26 | 2.37 | 0.43 | 0.96 | 12.15 | 0.51 | 0 | 4.38 |