E-commerce Agent Interaction on WebShop
70.4Average RewardMCTS
Evaluation Results
| Method | Links | |
|---|---|---|
| MCTSBase LLM=Llama2-70B2025.07 | 70.4 | |
| BPOApproach=System-2, Evaluation Protocol=Fine-Tuned, Base Model=Llama-3.1-8B-Instruct2025.08 | 67.45 | |
| Tree DPOBase LLM=Llama2-70B2025.07 | 65.9 | |
| SFTBase LLM=Llama2-70B2025.07 | 65.1 | |
| MPOApproach=System-2, Evaluation Protocol=Fine-Tuned, Base Model=Llama-3.1-8B-Instruct2025.08 | 55.2 | |
| Qwen-2.5-7B-InstructApproach=System-1, Evaluation Protocol=Zero-shot2025.08 | 54.28 | |
| SFTApproach=System-2, Evaluation Protocol=Fine-Tuned, Base Model=Llama-3.1-8B-Instruct2025.08 | 52.94 | |
| ETOApproach=System-2, Evaluation Protocol=Fine-Tuned, Base Model=Llama-3.1-8B-Instruct2025.08 | 52.08 | |
| o3-miniApproach=System-2, Evaluation Protocol=Zero-shot2025.08 | 49.02 | |
| Qwen-3-ThinkingApproach=System-2, Evaluation Protocol=Zero-shot2025.08 | 42.16 | |
| Deepseek-R1Approach=System-2, Evaluation Protocol=Zero-shot2025.08 | 40.31 | |
| Llama-3.1-8B-InstructApproach=System-1, Evaluation Protocol=Zero-shot2025.08 | 33.93 | |
| DELTAMEMLLM=DeepSeek-V4-flash, Decoding Temperature=0, Prompt Framework=ReAct, Node Embeddings=E5-base-v2, Learning Protocol=Online learning, in-context demonstration=nicl = 12026.06 | 0.6254 | |
| SynapseLLM=DeepSeek-V4-flash, Decoding Temperature=0, Prompt Framework=ReAct, Node Embeddings=E5-base-v2, Learning Protocol=Online learning, in-context demonstration=nicl = 12026.06 | 0.589 | |
| No MemoryLLM=DeepSeek-V4-flash, Decoding Temperature=0, Prompt Framework=ReAct, Node Embeddings=E5-base-v2, Learning Protocol=Zero-shot, in-context demonstration=nicl = 12026.06 | 0.3674 | |
| RBankLLM=DeepSeek-V4-flash, Decoding Temperature=0, Prompt Framework=ReAct, Node Embeddings=E5-base-v2, Learning Protocol=Online learning, in-context demonstration=nicl = 12026.06 | 0.3498 | |
| AWMLLM=DeepSeek-V4-flash, Decoding Temperature=0, Prompt Framework=ReAct, Node Embeddings=E5-base-v2, Learning Protocol=Online learning, in-context demonstration=nicl = 12026.06 | 0.3337 |