# Agentic Tool-use on tau2-bench Airline Source: [Agentic Tool-use on tau2-bench Airline]() Collection index: [llms.txt]() Data updated: 2026-05-25T07:34:35.536Z Evaluation of Agentic Tool-use performance on the Airline domain subset of the Tau2 benchmark, reporting pass rates at various thresholds (pass^1 through pass^4). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Agentic Tool-use | | Dataset (including any stated split) | tau2-bench Airline | | Metric | Pass@1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 34 of 34 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1](): Higher is better; unit: PERCENT. - [Pass@2](): Higher is better; unit: PERCENT. - [Pass@3](): Higher is better; unit: PERCENT. - [Pass@4](): Higher is better; unit: PERCENT. - [Score](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 | Result date | Report paper | | --- | --- | --- | --- | | CODEDELEGATOR | 63.5 | Not recorded | 6971af2ee25ad2fa9d1edad2 | | GPT-5 | 62.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | CODEDELEGATOR | 62 | Not recorded | 6971af2ee25ad2fa9d1edad2 | | CodeAct | 59 | Not recorded | 6971af2ee25ad2fa9d1edad2 | | ReAct | 58.5 | Not recorded | 6971af2ee25ad2fa9d1edad2 | | +PACE | 57.5 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | ReAct | 57 | Not recorded | 6971af2ee25ad2fa9d1edad2 | | CodeAct | 56.5 | Not recorded | 6971af2ee25ad2fa9d1edad2 | | +PE | 55 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | Vanilla | 53.5 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | Qwen3-235B-A22B-Inst. | 48.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | Simia-Tau-32B | 48.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | GPT-4o | 47.6 | Not recorded | 69a66c532d603e6de3e3d0a8 | | xLAM-2-32b-fc-r | 47 | Not recorded | 69a66c532d603e6de3e3d0a8 | | Sonnet 3.5 | 46 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | xLAM-2-70b-fc-r | 43.3 | Not recorded | 69a66c532d603e6de3e3d0a8 | | Simia-Tau-RL-8B | 43 | Not recorded | 69a66c532d603e6de3e3d0a8 | | CoVe-4B | 43 | Not recorded | 69a66c532d603e6de3e3d0a8 | | Simia-Tau-8B | 42.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | GPT-4o | 42 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | Simia-Tau-3B | 36 | Not recorded | 69a66c532d603e6de3e3d0a8 | | GEM-32B | 35.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | Qwen3-30B-A3B-Inst. | 33.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | xLAM-2-8b-fc-r | 33.5 | Not recorded | 69a66c532d603e6de3e3d0a8 | | +PACE | 27.5 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | Vanilla | 26.5 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | Qwen3-4B-Inst. | 25 | Not recorded | 69a66c532d603e6de3e3d0a8 | | xLAM-2-3b-fc-r | 24 | Not recorded | 69a66c532d603e6de3e3d0a8 | | +PE | 23.5 | Not recorded | 6a13d7886f71c5ae7dc9e89b | | GEM-8B | 22 | Not recorded | 69a66c532d603e6de3e3d0a8 | ### CODEDELEGATOR Pass@1: 63.5. Result ID: 6996767b5698665ec4bdf7dc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"DeepseekV3.2"}. This record does not assert independent reproduction. ### GPT-5 Pass@1: 62.5. Result ID: 69a85eff6dbb7075b94fe485. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Proprietary Models"}. This record does not assert independent reproduction. ### CODEDELEGATOR Pass@1: 62. Result ID: 6996767b5698665ec4bdf7df. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"GPT5.0"}. This record does not assert independent reproduction. ### CodeAct Pass@1: 59. Result ID: 6996767b5698665ec4bdf7db. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"DeepseekV3.2"}. This record does not assert independent reproduction. ### ReAct Pass@1: 58.5. Result ID: 6996767b5698665ec4bdf7da. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"DeepseekV3.2"}. This record does not assert independent reproduction. ### +PACE Pass@1: 57.5. Result ID: 6a13fb78fbc373592c0ad070. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"SLM":"Qwen3.5-9B"}. This record does not assert independent reproduction. ### ReAct Pass@1: 57. Result ID: 6996767b5698665ec4bdf7dd. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"GPT5.0"}. This record does not assert independent reproduction. ### CodeAct Pass@1: 56.5. Result ID: 6996767b5698665ec4bdf7de. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"GPT5.0"}. This record does not assert independent reproduction. ### +PE Pass@1: 55. Result ID: 6a13fb78fbc373592c0ad06f. [Reporting paper]() Recorded evaluation configuration: {"SLM":"Qwen3.5-9B"}. This record does not assert independent reproduction. ### Vanilla Pass@1: 53.5. Result ID: 6a13fb78fbc373592c0ad06e. [Reporting paper]() Recorded evaluation configuration: {"SLM":"Qwen3.5-9B"}. This record does not assert independent reproduction. ### Qwen3-235B-A22B-Inst. Pass@1: 48.5. Result ID: 69a85eff6dbb7075b94fe487. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≥70B"}. This record does not assert independent reproduction. ### Simia-Tau-32B Pass@1: 48.5. Result ID: 69a85f006dbb7075b94fe489. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"~30B"}. This record does not assert independent reproduction. ### GPT-4o Pass@1: 47.6. Result ID: 69a85eff6dbb7075b94fe486. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Proprietary Models"}. This record does not assert independent reproduction. ### xLAM-2-32b-fc-r Pass@1: 47. Result ID: 69a85f006dbb7075b94fe48a. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"~30B"}. This record does not assert independent reproduction. ### Sonnet 3.5 Pass@1: 46. Result ID: 6a13fb78fbc373592c0ad069. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### xLAM-2-70b-fc-r Pass@1: 43.3. Result ID: 69a85eff6dbb7075b94fe488. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≥70B"}. This record does not assert independent reproduction. ### Simia-Tau-RL-8B Pass@1: 43. Result ID: 69a85f006dbb7075b94fe48e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### CoVe-4B Pass@1: 43. Result ID: 69a85f006dbb7075b94fe494. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### Simia-Tau-8B Pass@1: 42.5. Result ID: 69a85f006dbb7075b94fe48d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### GPT-4o Pass@1: 42. Result ID: 6a13fb78fbc373592c0ad06a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Simia-Tau-3B Pass@1: 36. Result ID: 69a85f006dbb7075b94fe492. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### GEM-32B Pass@1: 35.5. Result ID: 69a85f006dbb7075b94fe48b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"~30B"}. This record does not assert independent reproduction. ### Qwen3-30B-A3B-Inst. Pass@1: 33.5. Result ID: 69a85f006dbb7075b94fe48c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"~30B"}. This record does not assert independent reproduction. ### xLAM-2-8b-fc-r Pass@1: 33.5. Result ID: 69a85f006dbb7075b94fe48f. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### +PACE Pass@1: 27.5. Result ID: 6a13fb78fbc373592c0ad06d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"SLM":"Qwen3-4B"}. This record does not assert independent reproduction. ### Vanilla Pass@1: 26.5. Result ID: 6a13fb78fbc373592c0ad06b. [Reporting paper]() Recorded evaluation configuration: {"SLM":"Qwen3-4B"}. This record does not assert independent reproduction. ### Qwen3-4B-Inst. Pass@1: 25. Result ID: 69a85f006dbb7075b94fe491. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### xLAM-2-3b-fc-r Pass@1: 24. Result ID: 69a85f006dbb7075b94fe493. [Reporting paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. ### +PE Pass@1: 23.5. Result ID: 6a13fb78fbc373592c0ad06c. [Reporting paper]() Recorded evaluation configuration: {"SLM":"Qwen3-4B"}. This record does not assert independent reproduction. ### GEM-8B Pass@1: 22. Result ID: 69a85f006dbb7075b94fe490. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Category":"Open-Source Models","Parameter Scale":"≤8B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.