Task and Scenario Goal Completion on AppWorld normal (test)
91.2Task Goal CompletionTrajectory-Informed Memory Generation
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Trajectory-Informed Memory GenerationMemory Level=Subtask-level tips, Selection Strategy=LLM-guided selection, Difficulty Level=Difficulty 12026.03 | 91.2 | 89.5 | — | — | |
| ASSAYModel=Sonnet 3.52026.06 | 89.3 | — | — | 5.4 | |
| ASSAYModel=GPT-4o-mini2026.06 | 88.7 | — | — | 6.6 | |
| wt-tunedModel=Qwen2-14B2026.06 | 86.9 | — | — | — | |
| ReActModel=Sonnet 3.52026.06 | 83.9 | — | — | — | |
| ASSAYModel=DeepSeek-V32026.06 | 83.3 | — | — | 14.2 | |
| ReActModel=GPT-4o-mini2026.06 | 82.1 | — | — | — | |
| ASSAYModel=Gemini 1.5 Pro2026.06 | 81 | — | — | 8.4 | |
| ACEModel=DeepSeek-V32026.06 | 78 | — | — | 8.9 | |
| ASSAYModel=GPT-3.52026.06 | 77.4 | — | — | 15.5 | |
| ASSAYModel=GPT-4-Turbo2026.06 | 75.6 | — | — | 8.9 | |
| Trajectory-Informed Memory GenerationMemory Level=Subtask-level tips, Selection Strategy=LLM-guided selection, Difficulty Level=Aggregate2026.03 | 73.2 | 64.3 | — | — | |
| CUGAModel=GPT-4-Turbo2026.06 | 73.2 | — | — | 6.5 | |
| ReActModel=Gemini 1.5 Pro2026.06 | 72.6 | — | — | — | |
| ASSAYModel=GPT-4o2026.06 | 71.4 | — | — | 22.6 | |
| Trajectory-Informed Memory GenerationMemory Level=Subtask-level tips, Selection Strategy=LLM-guided selection, Difficulty Level=Difficulty 22026.03 | 70.8 | 56.2 | — | — | |
| ReActModel=DeepSeek-V32026.06 | 69.1 | — | — | — | |
| Gupta et al.Model=GPT-4o2026.06 | 68.5 | — | — | 19.7 | |
| ACEModel=GPT-3.52026.06 | 67.3 | — | — | 5.4 | |
| ReActModel=GPT-4-Turbo2026.06 | 66.7 | — | — | — | |
| ReActModel=GPT-3.52026.06 | 61.9 | — | — | — | |
| ReAct + ACEBase LLM=GPT-OSS-120B, Adaptation Mode=Offline Adaptation, GT Labels=true2025.10 | 61.3 | 39.3 | — | — | |
| ReAct + ACEBase LLM=GPT-OSS-120B, Adaptation Mode=Online Adaptation, GT Labels=false2025.10 | 60.7 | 44.6 | — | — | |
| Trajectory-Informed Memory GenerationMemory Level=Subtask-level tips, Selection Strategy=LLM-guided selection, Difficulty Level=Difficulty 32026.03 | 58.7 | 47.6 | — | — | |
| ReAct + ACEBase LLM=GPT-OSS-120B, Adaptation Mode=Offline Adaptation, GT Labels=false2025.10 | 58.3 | 41.1 | — | — | |
| ReAct + GEPABase LLM=GPT-OSS-120B, Adaptation Mode=Offline Adaptation, GT Labels=true2025.10 | 56 | 33.9 | — | — | |
| ReActBase LLM=GPT-OSS-120B, Adaptation Mode=Base LLM2025.10 | 54.8 | 33.9 | — | — | |
| ReAct + DC (CU)Base LLM=GPT-OSS-120B, Adaptation Mode=Online Adaptation, GT Labels=false2025.10 | 49.4 | 33.9 | — | — | |
| GPT-4oParameters=—2026.04 | 48.8 | — | — | — | |
| ReActModel=GPT-4o2026.06 | 48.8 | — | — | — | |
| Claude 3.5 SonnetParameters=—2026.04 | 33.2 | — | — | — | |
| LLaMA-3-70B-InstructParameters=70B2026.04 | 20.8 | — | — | — | |
| Qwen3-8B + Scaffold (FP16)Parameters=8B, Precision=FP162026.04 | 8.9 | — | 5.5 | — | |
| DeepSeek-Coder 33B InstructParameters=33B2026.04 | 7.1 | — | 4.1 | — | |
| Qwen3-8B + Scaffold (AWQ)Parameters=8B, Precision=AWQ2026.04 | 5.9 | — | 3.3 | — | |
| Qwen3-8B Baseline (FP16)Parameters=8B, Precision=FP162026.04 | 5.4 | — | 2.8 | — | |
| Qwen3-8B Baseline (AWQ)Parameters=8B, Precision=AWQ2026.04 | 3 | — | 1.3 | — |