Tool-use Agent Evaluation on τ²-BENCH airline full 50-task pool
78Pass@4 Success RatePG-CHECKLIST
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| PG-CHECKLISTAgent=Sonnet 4.6, Variant=PG-CHECKLIST, Verifier=Sonnet 4.62026.06 | 78 | 100 | 57.7 | 6 | |
| PG-RAWAgent=Sonnet 4.6, Variant=PG-RAW, Verifier=Sonnet 4.62026.06 | 74 | 95.8 | 53.8 | 2 | |
| BaselineAgent=Sonnet 4.6, Variant=Baseline, Verifier=—2026.06 | 72 | 95.8 | 50 | — | |
| PG-CHECKLISTAgent=Gemini 2.5 Pro, Variant=PG-CHECKLIST, Verifier=Gemini 2.5 Pro2026.06 | 60 | 100 | 23.1 | 12 | |
| PG-CHECKLISTAgent=GPT 5.4, Variant=PG-CHECKLIST, Verifier=GPT 5.42026.06 | 58 | 100 | 19.2 | 12 | |
| TOOLGUARDAgent=Sonnet 4.6, Variant=TOOLGUARD, Verifier=Static code2026.06 | 58 | 100 | 19.2 | 14 | |
| TOOLGUARDAgent=GPT 5.4, Variant=TOOLGUARD, Verifier=Static code2026.06 | 52 | 87.5 | 19.2 | 6 | |
| PG-RAWAgent=Gemini 2.5 Pro, Variant=PG-RAW, Verifier=Gemini 2.5 Pro2026.06 | 50 | 75 | 26.9 | 2 | |
| PG-RAWAgent=GPT 5.4, Variant=PG-RAW, Verifier=GPT 5.42026.06 | 48 | 91.7 | 7.7 | 2 | |
| BaselineAgent=Gemini 2.5 Pro, Variant=Baseline, Verifier=—2026.06 | 48 | 75 | 23.1 | — | |
| BaselineAgent=GPT 5.4, Variant=Baseline, Verifier=—2026.06 | 46 | 75 | 19.2 | — | |
| TOOLGUARDAgent=Gemini 2.5 Pro, Variant=TOOLGUARD, Verifier=Static code2026.06 | 44 | 79.2 | 11.5 | 4 |