Agentic Task on AppWorld N (test)
57.1PR ScoreExpGraph
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ExpGraphExecutor LLM=Qwen3-32B (Small LLM)2026.05 | 57.1 | 8.8 | |
| ExpGraphExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM)2026.05 | 57.1 | 12.8 | |
| S3Executor LLM=Qwen3-32B (Small LLM), Method Strategy Category=LLM-Centric Experience Learning Baselines2026.05 | 55.4 | 9.7 | |
| S3Executor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=LLM-Centric Experience Learning Baselines2026.05 | 53 | 18.5 | |
| IRCoTExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=LLM-Centric Experience Learning Baselines2026.05 | 52.8 | 11.4 | |
| Search-o1Executor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=LLM-Centric Experience Learning Baselines2026.05 | 52.3 | 20.7 | |
| AWMExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 50.7 | 20.9 | |
| ReflexionExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Prompt-based Agentic Baselines2026.05 | 48.2 | 21.9 | |
| IRCoTExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=LLM-Centric Experience Learning Baselines2026.05 | 46.9 | 18.7 | |
| Search-o1Executor LLM=Qwen3-32B (Small LLM), Method Strategy Category=LLM-Centric Experience Learning Baselines2026.05 | 45.4 | 16.5 | |
| ReasoningBankExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 45.3 | 23.1 | |
| MemRLExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 45.2 | 18.6 | |
| ExpeLExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 43.6 | 21.1 | |
| MemRLExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 40.5 | 13.5 | |
| No-MemoryExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=No-Memory2026.05 | 34.2 | 33 | |
| LightMemExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 34.2 | 31.1 | |
| ReActExecutor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Prompt-based Agentic Baselines2026.05 | 32.7 | 24.8 | |
| Mem0Executor LLM=Gemini-3.1-Flash-Lite (Large LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 32.7 | 31.5 | |
| ReflexionExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Prompt-based Agentic Baselines2026.05 | 31.6 | 31.3 | |
| No-MemoryExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=No-Memory2026.05 | 30.1 | 30.8 | |
| ReActExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Prompt-based Agentic Baselines2026.05 | 30 | 32.2 | |
| ReasoningBankExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 29.4 | 25.6 | |
| LightMemExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 28.5 | 34.2 | |
| AWMExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 28 | 28.8 | |
| ExpeLExecutor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 26.7 | 35.1 | |
| Mem0Executor LLM=Qwen3-32B (Small LLM), Method Strategy Category=Retrieval-Centric Experience Learning Baselines2026.05 | 20.8 | 33.4 |