Agent Task on WebShop
99Success RateW2SG with MCTS
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| W2SG with MCTS2025.07 | 99 | — | 56.9 | — | — | — | — | |
| SFT Strong ModelBase Model=Llama-2-13b+ETO2025.07 | 97.5 | — | 52 | — | — | — | — | |
| W2SG with Tree DPO2025.07 | 97 | — | 53.2 | — | — | — | — | |
| Ceiling Model2025.07 | 96.5 | — | 58.3 | — | — | — | — | |
| SFT Strong Model + Best of N2025.07 | 96 | — | 52.3 | — | — | — | — | |
| SFT Strong ModelBase Model=Llama-2-13b+SFT2025.07 | 94 | — | 51 | — | — | — | — | |
| SFT Weak ModelBase Model=Llama-2-7b+SFT2025.07 | 87 | — | 47.1 | — | — | — | — | |
| Complementary RLexperience_retrieval_at_test_time=true2026.03 | 87 | — | — | — | — | — | — | |
| Complementary RLexperience_retrieval_at_test_time=false2026.03 | 84 | — | — | — | — | — | — | |
| Baseline2026.03 | 81 | — | — | — | — | — | — | |
| ResRLBackbone=Qwen2.5-7B-Instruct, Random seeds=32026.05 | 71.5 | — | — | 81.2 | — | — | — | |
| Q-EvolveBackbone=Llama-3-8B-Instruct2026.06 | 71.1 | — | — | — | — | — | — | |
| ETO + MPOBackbone=Llama-3-8B-Instruct2026.06 | 70.2 | — | — | — | — | — | — | |
| EMPGBackbone=Qwen2.5-7B-Instruct, Random seeds=32026.05 | 69.3 | — | — | 81 | — | — | — | |
| PPO (with critic)Backbone=Qwen2.5-7B-Instruct, Random seeds=32026.05 | 68.7 | — | — | 81.4 | — | — | — | |
| ETOBackbone=Llama-3-8B-Instruct2026.06 | 68.4 | — | — | — | — | — | — | |
| Static Online Exp.experience_retrieval_at_test_time=true2026.03 | 67 | — | — | — | — | — | — | |
| WKMBackbone=Llama-3-8B-Instruct2026.06 | 66.9 | — | — | — | — | — | — | |
| GRPOBackbone=Qwen2.5-7B-Instruct, Random seeds=32026.05 | 65.6 | — | — | 77.8 | — | — | — | |
| SFT + MPOBackbone=Llama-3-8B-Instruct2026.06 | 65.5 | — | — | — | — | — | — | |
| KnowAgentBackbone=Llama-3-8B-Instruct2026.06 | 64.8 | — | — | — | — | — | — | |
| SFTBackbone=Llama-3-8B-Instruct2026.06 | 63.3 | — | — | — | — | — | — | |
| Static Online Exp.experience_retrieval_at_test_time=false2026.03 | 59 | — | — | — | — | — | — | |
| Kintsugi2026.05 | 52 | — | — | 0.7392 | — | — | — | |
| Dual Memory2026.05 | 51 | — | — | 0.7132 | — | — | — | |
| LLMKnowledge Base=prompt2026.05 | 47 | — | — | 0.73 | — | — | — | |
| Gemini 2.5 ProModel Category=Thinking Model2026.02 | 43.2 | 27.1 | — | — | — | — | — | |
| DeepSeek-V3.2Model Category=Non-thinking Model2026.02 | 41 | 24.8 | — | — | — | — | — | |
| DeepSeek-R1Model Category=Thinking Model2026.02 | 38.4 | 24.1 | — | — | — | — | — | |
| Exp. Only2026.03 | 37 | — | — | — | — | — | — | |
| LLMKnowledge Base=tools2026.05 | 36 | — | — | 0.7034 | — | — | — | |
| Gemini-2.5-ProBackbone=Gemini-2.5-Pro, Random seeds=32026.05 | 35.9 | — | — | 42.5 | — | — | — | |
| Gemini 2.5 FlashModel Category=Non-thinking Model2026.02 | 35 | 21.6 | — | — | — | — | — | |
| Reflexion2026.05 | 35 | — | — | 0.5204 | — | — | — | |
| WALL-E 2.02026.05 | 34 | — | — | 0.5998 | — | — | — | |
| ReAct2026.05 | 32 | — | — | 0.501 | — | — | — | |
| ADaPT2026.05 | 32 | — | — | 0.5355 | — | — | — | |
| AWM2026.05 | 32 | — | — | 0.516 | — | — | — | |
| ExpeL2026.05 | 29 | — | — | 0.4582 | — | — | — | |
| Llama-3.3-70B-InstructModel Category=Non-thinking Model2026.02 | 26.8 | 17.6 | — | — | — | — | — | |
| gpt-oss-120bModel Category=Thinking Model2026.02 | 26.2 | 11.1 | — | — | — | — | — | |
| GLM-4-32B-0414Model Category=Non-thinking Model2026.02 | 25.4 | 17.2 | — | — | — | — | — | |
| GPT-4oBackbone=GPT-4o, Random seeds=32026.05 | 23.7 | — | — | 31.8 | — | — | — | |
| Phi-4-reasoningModel Category=Thinking Model2026.02 | 22.6 | 14.3 | — | — | — | — | — | |
| Qwen3-30B-A3B-ThinkingModel Category=Thinking Model2026.02 | 21 | 14.6 | — | — | — | — | — | |
| LLMKnowledge Base=None2026.05 | 21 | — | — | 0.5819 | — | — | — | |
| Qwen3-30B-A3B-InstructModel Category=Non-thinking Model2026.02 | 20.8 | 13.7 | — | — | — | — | — | |
| Llama-3.1-8B-InstructModel Category=Non-thinking Model2026.02 | 20.8 | 13.1 | — | — | — | — | — | |
| Prompting ReActBackbone=Qwen2.5-7B-Instruct, Random seeds=32026.05 | 19.5 | — | — | 46.2 | — | — | — | |
| Phi-4Model Category=Non-thinking Model2026.02 | 18.8 | 12.2 | — | — | — | — | — | |
| StateAct2026.05 | 17 | — | — | 0.2973 | — | — | — | |
| GLM-4-9B-ChatModel Category=Non-thinking Model2026.02 | 16.4 | 11.5 | — | — | — | — | — | |
| Ministral-3-14B-InstructModel Category=Non-thinking Model2026.02 | 15.2 | 5.4 | — | — | — | — | — | |
| Qwen3-4B-InstructModel Category=Non-thinking Model2026.02 | 15 | 9.6 | — | — | — | — | — | |
| Qwen3-4B-ThinkingModel Category=Thinking Model2026.02 | 12.2 | 8.2 | — | — | — | — | — | |
| Prompting BackboneBackbone=Qwen2.5-7B-Instruct, Random seeds=32026.05 | 7.8 | — | — | 26.4 | — | — | — | |
| Mistral-7B-InstructModel Category=Non-thinking Model2026.02 | 6.6 | 4.4 | — | — | — | — | — | |
| ADAMEMMemory Policy Setup=On-Policy, Backbone=Qwen2.5-7B-Instruct-RL, Long Term=Traj., Short Term=Strat.2026.06 | — | — | — | 74.2 | — | — | — | |
| ADAMEMMemory Policy Setup=Off-Policy, Backbone=Gemma-3-27b-it, Long Term=Traj., Short Term=Strat.2026.06 | — | — | — | 24.7 | — | — | — | |
| DAPOBackbone=Qwen3-4B2026.07 | — | — | — | — | 64.1 | 69.5 | 75 | |
| Gemini-2.5-ProBackbone=Gemini-2.5-Pro2026.07 | — | — | — | — | 37.6 | 39.7 | 40.4 | |
| GiGPOBackbone=Qwen3-4B2026.07 | — | — | — | — | 67.9 | 80.5 | 82.6 | |
| GPT-4oBackbone=GPT-4o2026.07 | — | — | — | — | 33.6 | 37.8 | 39.5 | |
| GRPOBackbone=Qwen3-4B2026.07 | — | — | — | — | 61.7 | 68.8 | 72.7 | |
| GSPOBackbone=Qwen3-4B2026.07 | — | — | — | — | 61.7 | 67.2 | 68.6 | |
| Mem0+GRPOBackbone=Qwen3-4B2026.07 | — | — | — | — | 63.1 | 69.5 | 74.8 | |
| MetaRLBackbone=Qwen3-4B2026.07 | — | — | — | — | 63.2 | 71.3 | 75.2 | |
| No MemoryMemory Policy Setup=On-Policy, Backbone=Qwen2.5-7B-Instruct-RL, Long Term=-, Short Term=-2026.06 | — | — | — | 71.4 | — | — | — | |
| No MemoryMemory Policy Setup=Off-Policy, Backbone=Gemma-3-27b-it, Long Term=-, Short Term=-2026.06 | — | — | — | 18.2 | — | — | — | |
| PivoARLBackbone=Qwen3-4B2026.07 | — | — | — | — | 66.7 | 81.5 | 83.8 | |
| ReActBackbone=Qwen3-4B2026.07 | — | — | — | — | 2.3 | 5.4 | 7.5 | |
| ReasoningBankMemory Policy Setup=On-Policy, Backbone=Qwen2.5-7B-Instruct-RL, Long Term=Strat., Short Term=Strat.2026.06 | — | — | — | 68.6 | — | — | — | |
| ReasoningBankMemory Policy Setup=Off-Policy, Backbone=Gemma-3-27b-it, Long Term=Strat., Short Term=Strat.2026.06 | — | — | — | 18.6 | — | — | — | |
| ReflexionBackbone=Qwen3-4B2026.07 | — | — | — | — | 1.55 | 3.18 | 5.1 | |
| SimpleMem+GRPOBackbone=Qwen3-4B2026.07 | — | — | — | — | 66.4 | 74.2 | 78.1 | |
| SKillRLBackbone=Qwen3-4B2026.07 | — | — | — | — | 68.2 | 72.6 | 74.2 | |
| SynapseMemory Policy Setup=On-Policy, Backbone=Qwen2.5-7B-Instruct-RL, Long Term=Traj., Short Term=Traj.2026.06 | — | — | — | 65.4 | — | — | — | |
| SynapseMemory Policy Setup=Off-Policy, Backbone=Gemma-3-27b-it, Long Term=Traj., Short Term=Traj.2026.06 | — | — | — | 22.6 | — | — | — |