Agent Tool-calling on Pokémon Red (20 held-out transitions)
50Action Applicability RateGemma-4 31B
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Gemma-4 31BStage=SFT2026.05 | 50 | 50 | 75 | 65 | 44 | |
| Gemma-4 26BStage=Offline GRPO (heuristic)2026.05 | 50 | 50 | 65 | 50 | 44 | |
| Gemma-4 26BStage=Offline GRPO (Gemini oracle)2026.05 | 50 | 50 | 55 | 30 | 40 | |
| Gemma-4 26BStage=Base2026.05 | 25 | 5 | 33 | 25 | 23 | |
| Qwen3.5 35BStage=Base2026.05 | 0 | — | — | — | — |