Agentic task solving on AppWorld
90TGCA-Evolve
Evaluation results
| Method | Links | ||
|---|---|---|---|
| A-EvolveBackbone=Claude Sonnet 4.52026.01 | 90 | 96.47 | |
| AWMBackbone=Claude Sonnet 4.52026.01 | 88 | 97.32 | |
| A-EvolveBackbone=GPT 52026.01 | 88 | 97.02 | |
| VanillaBackbone=Claude Sonnet 4.52026.01 | 86 | 94.94 | |
| AWMBackbone=GPT 52026.01 | 84 | 94.08 | |
| APEBackbone=GPT 52026.01 | 82 | 95 | |
| A-EvolveBackbone=Gemini 3 Flash2026.01 | 82 | 92.05 | |
| APEBackbone=Claude Sonnet 4.52026.01 | 80 | 85 | |
| VanillaBackbone=GPT 52026.01 | 80 | 94.15 | |
| Vanilla modelBase Model=DeepSeek-V3.2-Exp, Mem=N/A, Len=N/A2025.12 | 73.2 | — | |
| MNLBase Model=DeepSeek-V3.2-Exp, Mem=0, Len=02025.12 | 73.2 | — | |
| MementoBase Model=DeepSeek-V3.2-Exp, Mem=56, Len=6022025.12 | 64.2 | — | |
| A-EvolveBackbone=Claude Haiku 4.52026.01 | 64 | 84.31 | |
| A-EvolveBackbone=Claude Sonnet 42026.01 | 56 | 88.21 | |
| VanillaBackbone=Gemini 3 Flash2026.01 | 56 | 80.45 | |
| APEBackbone=Gemini 3 Flash2026.01 | 52 | 84 | |
| AWMBackbone=Gemini 3 Flash2026.01 | 52 | 87.75 | |
| AWMBackbone=Claude Sonnet 42026.01 | 50 | 86.29 | |
| AWMBackbone=Claude Haiku 4.52026.01 | 46 | 65.76 | |
| ACEBase Model=DeepSeek-V3.2-Exp, Mem=8, Len=69022025.12 | 44.6 | — | |
| APEBackbone=Claude Sonnet 42026.01 | 44 | 82 | |
| VanillaBackbone=Claude Sonnet 42026.01 | 42 | 79.42 | |
| VanillaBackbone=Claude Haiku 4.52026.01 | 32 | 51.16 | |
| APEBackbone=Claude Haiku 4.52026.01 | 30 | 56 | |
| MNLBase Model=Qwen3-8B, Mem=12, Len=3912025.12 | 14.3 | — | |
| Vanilla modelBase Model=Qwen3-8B, Mem=N/A, Len=N/A2025.12 | 12.5 | — | |
| MementoBase Model=Qwen3-8B, Mem=50, Len=7072025.12 | 12.5 | — | |
| ACEBase Model=Qwen3-8B, Mem=61, Len=32172025.12 | 0 | — |