# LLM Agent Evaluation on tau-bench Retail

Source: [LLM Agent Evaluation on tau-bench Retail](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-29T07:49:50.203Z

Detailed evaluation of LLM Agent capabilities on the Retail domain subset of the tau-bench dataset, reporting comprehensive metrics including success rates, efficiency, and policy compliance.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | LLM Agent Evaluation |
| Dataset (including any stated split) | tau-bench Retail |
| Metric | Pass@1 |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 48 of 48 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Pass@1](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Pass%401>): Higher is better; unit: PERCENT.

- [Success Rate](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Success%20Rate>): Higher is better; unit: PERCENT.

- [Pass@4](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Pass%404>): Higher is better; unit: PERCENT.

- [Avg Turns](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Avg%20Turns>): Lower is better; unit: SCALAR.

- [Avg Duration (s)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Avg%20Duration%20(s)>): Lower is better; unit: TIME.

- [Avg Tool Calls](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Avg%20Tool%20Calls>): Lower is better; unit: SCALAR.

- [Avg Tokens (K)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Avg%20Tokens%20(K)>): Lower is better; unit: SIZE.

- [Efficiency (Ieff)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Efficiency%20(Ieff)>): Higher is better; unit: PERCENT.

- [Burden Score](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Burden%20Score>): Lower is better; unit: SCALAR.

- [Verbosity Score](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Verbosity%20Score>): Lower is better; unit: SCALAR.

- [Intent Adherence](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Intent%20Adherence>): Higher is better; unit: PERCENT.

- [Question Fulfillment](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Question%20Fulfillment>): Higher is better; unit: PERCENT.

- [Policy Compliance](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Policy%20Compliance>): Higher is better; unit: PERCENT.

- [Policy Faithfulness](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Policy%20Faithfulness>): Higher is better; unit: PERCENT.

- [Execution Consistency](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Execution%20Consistency>): Higher is better; unit: PERCENT.

- [Data Faithfulness](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Data%20Faithfulness>): Higher is better; unit: PERCENT.

- [Missing Actions Rate](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Missing%20Actions%20Rate>): Lower is better; unit: PERCENT.

- [Identity Index](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Identity%20Index>): Higher is better; unit: SCALAR.

- [Interaction Policy Index](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Interaction%20Policy%20Index>): Higher is better; unit: SCALAR.

- [Tone Index](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Tone%20Index>): Higher is better; unit: SCALAR.

- [Accuracy](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Accuracy>): Higher is better; unit: PERCENT.

- [Delta Accuracy](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Delta%20Accuracy>): Higher is better; unit: SCALAR.

- [Score (S)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Score%20(S)>): Higher is better; unit: SCALAR.

- [Total Duration (Ttotal)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Total%20Duration%20(Ttotal)>): Lower is better; unit: TIME.

- [Delta Total Tokens (K)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Delta%20Total%20Tokens%20(K)>): Lower is better; unit: SIZE.

- [Task Duration (Ttask)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Task%20Duration%20(Ttask)>): Lower is better; unit: TIME.

- [Delta Task Tokens](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Delta%20Task%20Tokens>): Lower is better; unit: SIZE.

- [Steps Count (Tstep)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Steps%20Count%20(Tstep)>): Lower is better; unit: SCALAR.

- [Delta Step Tokens](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Delta%20Step%20Tokens>): Lower is better; unit: SIZE.

- [Pass@2](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Pass%402>): Higher is better; unit: PERCENT.

- [Pass@3](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Pass%403>): Higher is better; unit: PERCENT.

- [Pass@5](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-retail.md?metric=Pass%405>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Pass@1 | Result date | Report paper |
| --- | --- | --- | --- |
| EVOTOOL | 64.8 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| AdaPlanner | 62.2 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| EvoPrompt | 60.8 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| ANYTOOL | 60.4 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| ReAct | 59.8 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| Plan-and-Solve | 58.4 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| OPRO | 58.2 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| PromptBreeder | 56.4 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| EASYTOOL | 51.2 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| DRAFT | 48.4 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| FAMA | 44.17 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 43.47 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FAMA | 40.5 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 38.78 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 38 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FAMA | 37.9 | Not recorded | 69f190ad55a3c32af89fbb6e |
| EVOTOOL | 35.9 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| FC | 35 | Not recorded | 69f190ad55a3c32af89fbb6e |
| AdaPlanner | 34.8 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| FAMA | 34.6 | Not recorded | 69f190ad55a3c32af89fbb6e |
| CoT | 34.2 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| EvoPrompt | 33.8 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| ReAct | 33.1 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| FC | 32.6 | Not recorded | 69f190ad55a3c32af89fbb6e |
| Plan-and-Solve | 32 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| OPRO | 31.9 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| PromptBreeder | 30.6 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| IRMA | 28.86 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 28.5 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 26.9 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 25.2 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FC | 24.9 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ANYTOOL | 23.5 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| EASYTOOL | 20.9 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| FC | 19.65 | Not recorded | 69f190ad55a3c32af89fbb6e |
| DRAFT | 17.8 | Not recorded | 69aa5f6e2d603e6de3e417e0 |
| ReAct | 17.22 | Not recorded | 69f190ad55a3c32af89fbb6e |
| CoT | 14.6 | Not recorded | 69aa5f6e2d603e6de3e417e0 |

### EVOTOOL

Pass@1: 64.8. Result ID: 69aa8d216dbb7075b95438c9.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### AdaPlanner

Pass@1: 62.2. Result ID: 69aa8d216dbb7075b95438c5.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f971e209131cea467dd.md>)

[Implementation](<https://github.com/haotiansun14/AdaPlanner>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### EvoPrompt

Pass@1: 60.8. Result ID: 69aa8d216dbb7075b95438c4.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa785e1e209131cea7808c.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### ANYTOOL

Pass@1: 60.4. Result ID: 69aa8d216dbb7075b95438c8.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72141e209131cea545f6.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### ReAct

Pass@1: 59.8. Result ID: 69aa8d216dbb7075b95438bf.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### Plan-and-Solve

Pass@1: 58.4. Result ID: 69aa8d216dbb7075b95438c1.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7f1e209131cea45fd1.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### OPRO

Pass@1: 58.2. Result ID: 69aa8d216dbb7075b95438c2.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72f31e209131cea597cc.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### PromptBreeder

Pass@1: 56.4. Result ID: 69aa8d216dbb7075b95438c3.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70bb1e209131cea4ce79.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### EASYTOOL

Pass@1: 51.2. Result ID: 69aa8d216dbb7075b95438c6.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72b51e209131cea58113.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### DRAFT

Pass@1: 48.4. Result ID: 69aa8d216dbb7075b95438c7.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771e1e209131cea712ca.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### FAMA

Pass@1: 44.17. Result ID: 69f1b80847f34ce924d7f6bd.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### ReAct

Pass@1: 43.47. Result ID: 69f1b80847f34ce924d7f6ba.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### FAMA

Pass@1: 40.5. Result ID: 69f1b80847f34ce924d7f6b9.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### IRMA

Pass@1: 38.78. Result ID: 69f1b80847f34ce924d7f6bc.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### ReAct

Pass@1: 38. Result ID: 69f1b80847f34ce924d7f6b6.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### FAMA

Pass@1: 37.9. Result ID: 69f1b80847f34ce924d7f6b5.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### EVOTOOL

Pass@1: 35.9. Result ID: 69aa8d216dbb7075b95438d4.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### FC

Pass@1: 35. Result ID: 69f1b80847f34ce924d7f6b7.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### AdaPlanner

Pass@1: 34.8. Result ID: 69aa8d216dbb7075b95438d0.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f971e209131cea467dd.md>)

[Implementation](<https://github.com/haotiansun14/AdaPlanner>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### FAMA

Pass@1: 34.6. Result ID: 69f1b80847f34ce924d7f6b1.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### CoT

Pass@1: 34.2. Result ID: 69aa8d216dbb7075b95438c0.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e571e209131cea3f27b.md>)

Recorded evaluation configuration: {"Backbone":"GPT-4o"}. This record does not assert independent reproduction.

### EvoPrompt

Pass@1: 33.8. Result ID: 69aa8d216dbb7075b95438cf.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa785e1e209131cea7808c.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### ReAct

Pass@1: 33.1. Result ID: 69aa8d216dbb7075b95438ca.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### FC

Pass@1: 32.6. Result ID: 69f1b80847f34ce924d7f6b3.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### Plan-and-Solve

Pass@1: 32. Result ID: 69aa8d216dbb7075b95438cc.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f7f1e209131cea45fd1.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### OPRO

Pass@1: 31.9. Result ID: 69aa8d216dbb7075b95438cd.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72f31e209131cea597cc.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### PromptBreeder

Pass@1: 30.6. Result ID: 69aa8d216dbb7075b95438ce.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70bb1e209131cea4ce79.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### IRMA

Pass@1: 28.86. Result ID: 69f1b80847f34ce924d7f6b0.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### IRMA

Pass@1: 28.5. Result ID: 69f1b80847f34ce924d7f6b4.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### IRMA

Pass@1: 26.9. Result ID: 69f1b80847f34ce924d7f6b8.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### ReAct

Pass@1: 25.2. Result ID: 69f1b80847f34ce924d7f6b2.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### FC

Pass@1: 24.9. Result ID: 69f1b80847f34ce924d7f6af.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### ANYTOOL

Pass@1: 23.5. Result ID: 69aa8d216dbb7075b95438d3.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72141e209131cea545f6.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### EASYTOOL

Pass@1: 20.9. Result ID: 69aa8d216dbb7075b95438d1.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72b51e209131cea58113.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### FC

Pass@1: 19.65. Result ID: 69f1b80847f34ce924d7f6bb.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### DRAFT

Pass@1: 17.8. Result ID: 69aa8d216dbb7075b95438d2.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771e1e209131cea712ca.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### ReAct

Pass@1: 17.22. Result ID: 69f1b80847f34ce924d7f6ae.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### CoT

Pass@1: 14.6. Result ID: 69aa8d216dbb7075b95438cb.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f6e2d603e6de3e417e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e571e209131cea3f27b.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.