Task Failure Prediction and Selective Completion on tau2-bench Airline 1.0
74.2AUROCTRACER
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| TRACERModel=gpt-4.1-mini2026.02 | 74.2 | 61.5 | |
| TRACERModel=gemini-2.5-pro2026.02 | 73.5 | 62.9 | |
| TRACERModel=gemini-2.5-flash2026.02 | 72.5 | 69.7 | |
| Semantic EntropyModel=gemini-2.5-flash2026.02 | 66.6 | 64.8 | |
| Self-Assessed ConfidenceModel=gemini-2.5-flash2026.02 | 66.3 | 64.5 | |
| Semantic EntropyModel=gemini-2.5-pro2026.02 | 60.3 | 51.6 | |
| Normalized EntropyModel=gemini-2.5-pro2026.02 | 60 | 51.4 | |
| SAUPModel=gemini-2.5-pro2026.02 | 59.5 | 51.7 | |
| Normalized EntropyModel=gemini-2.5-flash2026.02 | 56.8 | 61.2 | |
| Semantic EntropyModel=gpt-4.1-mini2026.02 | 54.1 | 42.7 | |
| SAUPModel=gemini-2.5-flash2026.02 | 54 | 59.9 | |
| Normalized EntropyModel=gpt-4.1-mini2026.02 | 53.8 | 40.2 | |
| SAUPModel=gpt-4.1-mini2026.02 | 53.1 | 40.3 | |
| Self-Assessed ConfidenceModel=gemini-2.5-pro2026.02 | 46.2 | 45.1 | |
| Self-Assessed ConfidenceModel=gpt-4.1-mini2026.02 | 29 | 23.6 |