Task goal completion on AppWorld (test challenge)
32Goal Completion ScoreRLOO
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| RLOOBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 32 | 40 | 0.01 | |
| GSPOBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 32 | 36 | 0 | |
| ADAPOBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 30 | 33 | 0 | |
| DAPOBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 28 | 29 | 0.06 | |
| GRPO + H bonusBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 26 | 28 | -0.01 | |
| REPO-RBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 25 | 27 | -0.01 | |
| GRPOBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 21 | 23 | -0.09 | |
| LOOPBase Model=Qwen-3-8B, Max Context Length=40962026.03 | 19 | 22 | -0.08 | |
| ASSAYModel=GPT-4o-mini2026.06 | 0.854 | — | 4.3 | |
| ReActModel=GPT-4o-mini2026.06 | 0.811 | — | — | |
| ASSAYModel=Sonnet 3.52026.06 | 0.753 | — | 5 | |
| ReActModel=Sonnet 3.52026.06 | 0.703 | — | — | |
| ASSAYModel=DeepSeek-V32026.06 | 0.693 | — | 22.3 | |
| wt-tunedModel=Qwen2-14B2026.06 | 0.676 | — | — | |
| ASSAYModel=GPT-3.52026.06 | 0.664 | — | 13.9 | |
| ASSAYModel=GPT-4-Turbo2026.06 | 0.64 | — | 13.6 | |
| ACEModel=DeepSeek-V32026.06 | 0.631 | — | 16.1 | |
| RLOOBase Model=Qwen-3-32B2026.03 | 0.61 | 0.71 | -0.18 | |
| ADAPOBase Model=Qwen-3-32B2026.03 | 0.58 | 0.62 | 0.52 | |
| CUGAModel=GPT-4-Turbo2026.06 | 0.576 | — | 7.2 | |
| ASSAYModel=Gemini 1.5 Pro2026.06 | 0.549 | — | 5.5 | |
| REPO-RBase Model=Qwen-3-32B2026.03 | 0.54 | 0.63 | 0.03 | |
| ReActModel=GPT-3.52026.06 | 0.525 | — | — | |
| DAPOBase Model=Qwen-3-32B2026.03 | 0.52 | 0.55 | 1.52 | |
| GSPOBase Model=Qwen-3-32B2026.03 | 0.51 | 0.51 | -0.07 | |
| ReActModel=GPT-4-Turbo2026.06 | 0.504 | — | — | |
| ACEModel=GPT-3.52026.06 | 0.499 | — | 2.6 | |
| ReActModel=Gemini 1.5 Pro2026.06 | 0.494 | — | — | |
| GRPO + H bonusBase Model=Qwen-3-32B2026.03 | 0.49 | 0.51 | 0.14 | |
| ReActModel=DeepSeek-V32026.06 | 0.47 | — | — | |
| GRPOBase Model=Qwen-3-32B2026.03 | 0.46 | 0.49 | -0.29 | |
| LOOPBase Model=Qwen-3-32B2026.03 | 0.45 | 0.47 | -0.31 | |
| ASSAYModel=GPT-4o2026.06 | 0.41 | — | 10.8 | |
| Gupta et al.Model=GPT-4o2026.06 | 0.389 | — | 8.7 | |
| ReActModel=GPT-4o2026.06 | 0.302 | — | — |