# LLM Agent Evaluation on tau-bench Airline

Source: [LLM Agent Evaluation on tau-bench Airline](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-27T10:13:01.055Z

Comprehensive evaluation of LLM Agent performance on the tau-bench Airline domain, measuring success rate, efficiency (duration, tool calls), and adherence to defined policies and intent.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | LLM Agent Evaluation |
| Dataset (including any stated split) | tau-bench Airline |
| Metric | Pass@4 |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 36 of 36 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Pass@4](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Pass%404>): Higher is better; unit: PERCENT.

- [Success Rate](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Success%20Rate>): Higher is better; unit: PERCENT.

- [Avg Turns](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Avg%20Turns>): Lower is better; unit: SCALAR.

- [Avg Duration (s)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Avg%20Duration%20(s)>): Lower is better; unit: TIME.

- [Avg Tool Calls](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Avg%20Tool%20Calls>): Lower is better; unit: SCALAR.

- [Avg Tokens (K)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Avg%20Tokens%20(K)>): Lower is better; unit: SCALAR.

- [Efficiency (Ieff)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Efficiency%20(Ieff)>): Higher is better; unit: PERCENT.

- [Burden (B(tau))](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Burden%20(B(tau))>): Lower is better; unit: SCALAR.

- [Verbosity (V(tau))](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Verbosity%20(V(tau))>): Lower is better; unit: SCALAR.

- [Intent Adherence (Iintent)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Intent%20Adherence%20(Iintent)>): Higher is better; unit: PERCENT.

- [Query Fulfillment (Iqf)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Query%20Fulfillment%20(Iqf)>): Higher is better; unit: PERCENT.

- [Policy Compliance (Ipc)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Policy%20Compliance%20(Ipc)>): Higher is better; unit: PERCENT.

- [Policy Faithfulness (Ipf)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Policy%20Faithfulness%20(Ipf)>): Higher is better; unit: PERCENT.

- [Execution Consistency (Iec)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Execution%20Consistency%20(Iec)>): Higher is better; unit: PERCENT.

- [Data Faithfulness (Idf)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Data%20Faithfulness%20(Idf)>): Higher is better; unit: PERCENT.

- [Missing Actions Rate](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Missing%20Actions%20Rate>): Lower is better; unit: PERCENT.

- [IID Score](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=IID%20Score>): Higher is better; unit: SCALAR.

- [IPII Score](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=IPII%20Score>): Higher is better; unit: SCALAR.

- [ITone Score](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=ITone%20Score>): Higher is better; unit: SCALAR.

- [Accuracy](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Accuracy>): Higher is better; unit: PERCENT.

- [Delta Accuracy (ΔAcc)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Delta%20Accuracy%20(%CE%94Acc)>): Higher is better; unit: SCALAR.

- [Score S](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Score%20S>): Higher is better; unit: SCALAR.

- [Total Tokens (K)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Total%20Tokens%20(K)>): Higher is better; unit: SCALAR.

- [Delta Total Tokens (K)](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Delta%20Total%20Tokens%20(K)>): Higher is better; unit: SCALAR.

- [Task Duration](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Task%20Duration>): Lower is better; unit: TIME.

- [Delta Task Tokens](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Delta%20Task%20Tokens>): Higher is better; unit: SCALAR.

- [Total Steps](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Total%20Steps>): Lower is better; unit: SCALAR.

- [Delta Step Tokens](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Delta%20Step%20Tokens>): Higher is better; unit: SCALAR.

- [Pass@1](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Pass%401>): Higher is better; unit: PERCENT.

- [Pass@2](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Pass%402>): Higher is better; unit: PERCENT.

- [Pass@3](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Pass%403>): Higher is better; unit: PERCENT.

- [Pass@5](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Pass%405>): Higher is better; unit: PERCENT.

- [Average Score @4](<https://www.sota2.com/research/sota/llm-agent-evaluation-on-tau-bench-airline.md?metric=Average%20Score%20%404>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Pass@4 | Result date | Report paper |
| --- | --- | --- | --- |
| NoisyAgent | 78 | Not recorded | 6a167ad3dbd7150506f80299 |
| DAPO | 76 | Not recorded | 6a167ad3dbd7150506f80299 |
| GSPO | 74 | Not recorded | 6a167ad3dbd7150506f80299 |
| GRPO | 72 | Not recorded | 6a167ad3dbd7150506f80299 |
| NoisyAgent | 68 | Not recorded | 6a167ad3dbd7150506f80299 |
| DAPO | 66 | Not recorded | 6a167ad3dbd7150506f80299 |
| Qwen3-32B | 66 | Not recorded | 6a167ad3dbd7150506f80299 |
| GSPO | 64 | Not recorded | 6a167ad3dbd7150506f80299 |
| GRPO | 62 | Not recorded | 6a167ad3dbd7150506f80299 |
| Qwen3-8B | 52 | Not recorded | 6a167ad3dbd7150506f80299 |
| GPT-5 | 44 | Not recorded | 69a7bc872d603e6de3e3d56d |
| Kimi-K2 | 28 | Not recorded | 69a7bc872d603e6de3e3d56d |
| FAMA | 26.7 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 26.4 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 19.6 | Not recorded | 69f190ad55a3c32af89fbb6e |
| Mistral | 18 | Not recorded | 69a7bc872d603e6de3e3d56d |
| FAMA | 18 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FAMA | 18 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FAMA | 16.6 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FC | 15.2 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 14.8 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 13.2 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 12.8 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 11.6 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FC | 10.8 | Not recorded | 69f190ad55a3c32af89fbb6e |
| IRMA | 10 | Not recorded | 69f190ad55a3c32af89fbb6e |
| ReAct | 8.4 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FC | 8 | Not recorded | 69f190ad55a3c32af89fbb6e |
| FC | 2.4 | Not recorded | 69f190ad55a3c32af89fbb6e |

### NoisyAgent

Pass@4: 78. Result ID: 6a16c399fbc373592c1080da.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-32B"}. This record does not assert independent reproduction.

### DAPO

Pass@4: 76. Result ID: 6a16c399fbc373592c1080d8.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78a81e209131cea79ac6.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-32B"}. This record does not assert independent reproduction.

### GSPO

Pass@4: 74. Result ID: 6a16c399fbc373592c1080d9.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a481e209131cea8231d.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-32B"}. This record does not assert independent reproduction.

### GRPO

Pass@4: 72. Result ID: 6a16c399fbc373592c1080d7.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"backbone":"Qwen3-32B"}. This record does not assert independent reproduction.

### NoisyAgent

Pass@4: 68. Result ID: 6a16c399fbc373592c1080d5.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### DAPO

Pass@4: 66. Result ID: 6a16c399fbc373592c1080d3.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78a81e209131cea79ac6.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### Qwen3-32B

Pass@4: 66. Result ID: 6a16c399fbc373592c1080d6.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-32B"}. This record does not assert independent reproduction.

### GSPO

Pass@4: 64. Result ID: 6a16c399fbc373592c1080d4.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a481e209131cea8231d.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### GRPO

Pass@4: 62. Result ID: 6a16c399fbc373592c1080d2.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### Qwen3-8B

Pass@4: 52. Result ID: 6a16c399fbc373592c1080d1.

[Reporting paper](<https://www.sota2.com/research/paper/6a167ad3dbd7150506f80299.md>)

Recorded evaluation configuration: {"backbone":"Qwen3-8B"}. This record does not assert independent reproduction.

### GPT-5

Pass@4: 44. Result ID: 69a89f286dbb7075b9518fab.

[Reporting paper](<https://www.sota2.com/research/paper/69a7bc872d603e6de3e3d56d.md>)

Recorded evaluation configuration: {"k":"4"}. This record does not assert independent reproduction.

### Kimi-K2

Pass@4: 28. Result ID: 69a89f286dbb7075b9518fac.

[Reporting paper](<https://www.sota2.com/research/paper/69a7bc872d603e6de3e3d56d.md>)

Recorded evaluation configuration: {"k":"4"}. This record does not assert independent reproduction.

### FAMA

Pass@4: 26.7. Result ID: 69f1b80847f34ce924d7f6a1.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### ReAct

Pass@4: 26.4. Result ID: 69f1b80847f34ce924d7f69e.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### IRMA

Pass@4: 19.6. Result ID: 69f1b80847f34ce924d7f6a4.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### Mistral

Pass@4: 18. Result ID: 69a89f286dbb7075b9518fad.

[Reporting paper](<https://www.sota2.com/research/paper/69a7bc872d603e6de3e3d56d.md>)

Recorded evaluation configuration: {"k":"4"}. This record does not assert independent reproduction.

### FAMA

Pass@4: 18. Result ID: 69f1b80847f34ce924d7f6a9.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### FAMA

Pass@4: 18. Result ID: 69f1b80847f34ce924d7f6ad.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### FAMA

Pass@4: 16.6. Result ID: 69f1b80847f34ce924d7f6a5.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### FC

Pass@4: 15.2. Result ID: 69f1b80847f34ce924d7f69f.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### ReAct

Pass@4: 14.8. Result ID: 69f1b80847f34ce924d7f6a6.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### IRMA

Pass@4: 13.2. Result ID: 69f1b80847f34ce924d7f6a0.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-4B-Instruct"}. This record does not assert independent reproduction.

### ReAct

Pass@4: 12.8. Result ID: 69f1b80847f34ce924d7f6aa.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### IRMA

Pass@4: 11.6. Result ID: 69f1b80847f34ce924d7f6ac.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

### FC

Pass@4: 10.8. Result ID: 69f1b80847f34ce924d7f6a7.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### IRMA

Pass@4: 10. Result ID: 69f1b80847f34ce924d7f6a8.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b091e209131cea86582.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-32B"}. This record does not assert independent reproduction.

### ReAct

Pass@4: 8.4. Result ID: 69f1b80847f34ce924d7f6a2.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6eb41e209131cea41992.md>)

[Implementation](<https://react-lm.github.io/>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### FC

Pass@4: 8. Result ID: 69f1b80847f34ce924d7f6a3.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen3-14B"}. This record does not assert independent reproduction.

### FC

Pass@4: 2.4. Result ID: 69f1b80847f34ce924d7f6ab.

[Reporting paper](<https://www.sota2.com/research/paper/69f190ad55a3c32af89fbb6e.md>)

Recorded evaluation configuration: {"Backbone Model":"Qwen2.5-72B-Instruct"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.