Agentic Tool-use on AppWorld (Challenge)
83.7TGCRCL (all primitives)
Evaluation results
| Method | Links | |
|---|---|---|
| RCL (all primitives)Agent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 83.7 | |
| ACE + BatchingAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 83.4 | |
| ACE + Failure ReplayAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 81.8 | |
| ACE + Grouped RolloutsAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 81.4 | |
| ACEAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 80.7 | |
| ACE + Auxiliary LossesAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 80.6 | |
| ACE + Optimizer StateAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 79.4 | |
| ACE + Credit AssignmentAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 79.4 | |
| SeedAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 75.8 | |
| ACE + Auxiliary LossesAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 74.6 | |
| ACE + Optimizer StateAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 73.9 | |
| ACE + Failure ReplayAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 73.1 | |
| ACE + Grouped RolloutsAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 72.4 | |
| RCL (all primitives)Agent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 71.9 | |
| ACE + Credit AssignmentAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 71.5 | |
| ACE + BatchingAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 71.2 | |
| ACEAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 69.1 | |
| GEPAAgent Model=GPT-5.4 Nano, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 66.7 | |
| SeedAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 64.3 | |
| GEPAAgent Model=Gemini 3.1 Flash Lite, Optimizer Model=Claude Opus, Optimization Iterations=302026.04 | 62.6 |