Agent task completion on AppWorld
83.7TGC Success Rate (N)PREPING
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| PREPINGMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 83.7 | 70.2 | 72.2 | 54.7 | 70.2 | |
| ACE-OfflineMethod Category=Task-Informed Methods, Base LLM=DeepSeek-V3.22026.05 | 82.7 | 69.1 | 69 | 50.3 | 67.8 | |
| ACE-OnlineMethod Category=Task-Informed Methods, Base LLM=DeepSeek-V3.22026.05 | 80.4 | 65.5 | 78.3 | 60.9 | 71.3 | |
| Guided ExplorationMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 74.6 | 56 | 56.6 | 35 | 55.6 | |
| Random ExplorationMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 73.6 | 52.4 | 55.9 | 36.9 | 54.7 | |
| Direct MemoryMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 72.8 | 55.4 | 56 | 34.5 | 54.7 | |
| BaseMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 69.6 | 49.4 | 56.7 | 36.7 | 53.1 |