E-commerce product search and purchase on WebShop
81.3Strict SuccessSIRI
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SIRICategory=Memory-Augmented RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 81.3 | — | 89.9 | |
| TRIAGEModel=Qwen2.5-7B-Instruct, Judge=Qwen3-8B-thinking, Evidence=true2026.06 | 77.2 | — | — | |
| TRIAGEModel=Qwen2.5-7B-Instruct, Judge=Qwen3-8B-thinking, Evidence=false2026.06 | 73.5 | — | — | |
| GiGPOCategory=RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 72.8 | — | 84.4 | |
| SkillRLCategory=Memory-Augmented RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 72.7 | — | 85.2 | |
| GRPOModel=Qwen2.5-7B-Instruct2026.06 | 70.1 | — | — | |
| PPOCategory=RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 68.7 | — | 81.4 | |
| GRPOCategory=RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 66.1 | — | 79.3 | |
| RLOOCategory=RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 65.7 | — | 80.3 | |
| TRIAGEModel=Qwen2.5-7B-Instruct, Judge=Qwen3-8B no-think2026.06 | 65.4 | — | — | |
| TRIAGEModel=Qwen3-1.7B-Instruct, Judge=Qwen3-8B-thinking, Evidence=true2026.06 | 55.9 | — | — | |
| TRIAGEModel=Qwen3-1.7B-Instruct, Judge=Qwen3-8B-thinking, Evidence=false2026.06 | 49.6 | — | — | |
| SimpleMem+GRPOCategory=Memory-Augmented RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 46.9 | — | 67.8 | |
| Mem0+GRPOCategory=Memory-Augmented RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 37.5 | — | 58.1 | |
| GRPOModel=Qwen3-1.7B-Instruct2026.06 | 37.5 | — | — | |
| ActFocusBackbone=3B, Evaluation Step=1002026.05 | 36.7 | 99.2 | — | |
| TRIAGEModel=Qwen3-1.7B-Instruct, Judge=Qwen3-8B no-think2026.06 | 35.1 | — | — | |
| ReflexionCategory=Prompt-based Agentic or Memory-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 28.8 | — | 58.1 | |
| ActFocusBackbone=1.5B, Evaluation Step=1002026.05 | 26.6 | 98.8 | — | |
| UniformBackbone=3B, Evaluation Step=1002026.05 | 22.7 | 96.9 | — | |
| ReActCategory=Prompt-based Agentic or Memory-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 19.5 | — | 46.2 | |
| EvolverCategory=Memory-Augmented RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 17.6 | — | 42.5 | |
| Gemini-3-FlashCategory=Closed-source LLMs2026.06 | 16.5 | — | 14.1 | |
| GLM-5.1Category=Closed-source LLMs2026.06 | 12.5 | — | 15.6 | |
| UniformBackbone=1.5B, Evaluation Step=1002026.05 | 9.8 | 94.9 | — | |
| MemRLCategory=Memory-Augmented RL-based Methods, Backbone=Qwen2.5-7B-Instruct2026.06 | 9.2 | — | 29.5 | |
| OriginBackbone=Qwen2.5-7B-Instruct2026.06 | 3.9 | — | 16.6 |