Sequential environment decision making on Crafter BALROG protocol
57.3Peak Task Score (%)Gemini-3-Pro
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemini-3-Procategory=Frontier proprietary (BALROG leaderboard)2026.07 | 57.3 | — | |
| Gemini-3.1-Pro-Thinkingcategory=Frontier proprietary (BALROG leaderboard)2026.07 | 55 | — | |
| Gemini-2.5-Procategory=Frontier proprietary (BALROG leaderboard)2026.07 | 55 | — | |
| AUTOMEMbackbone=Qwen2.5-32B-Instruct, setup=memory training (loop #2)2026.07 | 51.36 | — | |
| Claude-Opus-4.5category=Frontier proprietary (BALROG leaderboard)2026.07 | 49.5 | — | |
| AUTOMEMbackbone=Qwen2.5-32B-Instruct, setup=scaffold opt. (loop #1)2026.07 | 47.27 | — | |
| MAGETeacher=Opus, n=52026.05 | 37.9 | — | |
| MAGE Opus–Llama (n=5, peak)Iters=52026.05 | 37.9 | 4 | |
| DeepSeek-R1category=Open-weight (BALROG leaderboard)2026.07 | 36.4 | — | |
| MAGETeacher=Haiku, n=52026.05 | 34.8 | — | |
| MAGE Haiku–Llama (n=5, peak)Iters=52026.05 | 34.8 | 4 | |
| MAGETeacher=Sonnet, n=52026.05 | 33.5 | — | |
| MAGE Sonnet–Llama (n=4, peak)Iters=52026.05 | 33.5 | 4 | |
| Qwen2.5-72B-Instructcategory=Open-weight (BALROG leaderboard)2026.07 | 27.3 | — | |
| Standalone Llama-3.1-8BProtocol=BALROG, seeds=32026.05 | 25.5 | — | |
| Standalone Llama-3.1-8B (BALROG ref.)Iters=12026.05 | 25.5 | — | |
| AUTOMEMbackbone=Qwen2.5-32B-Instruct, setup=memory-as-file-system, v02026.07 | 25 | — | |
| Qwen2.5-32B-Instructcontext-management method=sliding window2026.07 | 19.55 | — | |
| Qwen2.5-32B-Instructcontext-management method=sliding window + chain-of-thought2026.07 | 17.27 | — | |
| Qwen2.5-7B-Instructcategory=Open-weight (BALROG leaderboard)2026.07 | 16.4 | — |