Tool-calling on AppWorld (test-normal)
45.6Task Goal Completion (TGC)SGCD
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SGCDBackbone=Qwen3.5-4B, Decoding strategy=greedy pass@12026.06 | 45.6 | 17.9 | |
| GRPOBackbone=Qwen3.5-4B, Decoding strategy=greedy pass@12026.06 | 42.9 | 16.1 | |
| KATEModel=Qwen3-32B2026.06 | 32.7 | 10.7 | |
| ReAct + STModel=Qwen3-32B2026.06 | 27.4 | 1.8 | |
| KATEModel=Qwen3-8B2026.06 | 26.8 | 10.7 | |
| ReAct + STModel=Qwen3-8B2026.06 | 26.2 | 10.7 | |
| Base Qwen3.5-4BBackbone=Qwen3.5-4B, Decoding strategy=greedy pass@12026.06 | 23.8 | 10.7 | |
| MempModel=Qwen3-32B2026.06 | 22.6 | 5.4 | |
| MempModel=Qwen3-8B2026.06 | 22 | 7.1 | |
| ReActModel=Qwen3-32B2026.06 | 16.7 | 1.8 | |
| ReActModel=Qwen3-8B2026.06 | 10.1 | 1.8 |