Web Agent Task Success on WebArena Shopping 187 tasks
54Task Success RateReasoningBank + MaTTS
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ReasoningBank + MaTTSBackbone LLM=Gemini-2.5-pro, k=5, sampling=pass@12025.09 | 54 | 5.9 | |
| ReasoningBank + MaTTSBackbone LLM=Gemini-2.5-flash, k=5, sampling=pass@12025.09 | 53 | 6.3 | |
| ReasoningBankBackbone LLM=Gemini-2.5-pro, k=5, sampling=pass@12025.09 | 51.9 | 6 | |
| ReasoningBankBackbone LLM=Gemini-2.5-flash, k=5, sampling=pass@12025.09 | 49.7 | 6.1 | |
| AWMBackbone LLM=Gemini-2.5-pro, k=5, sampling=pass@12025.09 | 48.1 | 6.4 | |
| ReasoningBank + MaTTSBackbone LLM=Claude-3.7-sonnet, k=5, sampling=pass@12025.09 | 47.1 | 5.8 | |
| SynapseBackbone LLM=Gemini-2.5-pro, k=5, sampling=pass@12025.09 | 46.5 | 6.6 | |
| No MemoryBackbone LLM=Gemini-2.5-pro, k=5, sampling=pass@12025.09 | 45.5 | 7.6 | |
| ReasoningBankBackbone LLM=Claude-3.7-sonnet, k=5, sampling=pass@12025.09 | 44.9 | 5.6 | |
| AWMBackbone LLM=Gemini-2.5-flash, k=5, sampling=pass@12025.09 | 44.4 | 7 | |
| SynapseBackbone LLM=Gemini-2.5-flash, k=5, sampling=pass@12025.09 | 40.6 | 7 | |
| SynapseBackbone LLM=Claude-3.7-sonnet, k=5, sampling=pass@12025.09 | 39.6 | 5.8 | |
| AWMBackbone LLM=Claude-3.7-sonnet, k=5, sampling=pass@12025.09 | 39.6 | 7.2 | |
| No MemoryBackbone LLM=Gemini-2.5-flash, k=5, sampling=pass@12025.09 | 39 | 8.2 | |
| No MemoryBackbone LLM=Claude-3.7-sonnet, k=5, sampling=pass@12025.09 | 38.5 | 6.1 |