Task failure prediction and selective task completion on tau2-bench Retail 1.0
0.707AUROCTRACER
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| TRACERModel=gemini-2.5-flash2026.02 | 0.707 | 0.67 | |
| TRACERModel=gpt-4.1-mini2026.02 | 0.689 | 0.632 | |
| TRACERModel=gemini-2.5-pro2026.02 | 0.673 | 0.725 | |
| Semantic EntropyModel=gpt-4.1-mini2026.02 | 0.62 | 0.579 | |
| Normalized EntropyModel=gpt-4.1-mini2026.02 | 0.617 | 0.571 | |
| SAUPModel=gpt-4.1-mini2026.02 | 0.609 | 0.584 | |
| Self-Assessed ConfidenceModel=gpt-4.1-mini2026.02 | 0.577 | 0.555 | |
| Semantic EntropyModel=gemini-2.5-pro2026.02 | 0.556 | 0.673 | |
| Normalized EntropyModel=gemini-2.5-pro2026.02 | 0.553 | 0.661 | |
| Semantic EntropyModel=gemini-2.5-flash2026.02 | 0.533 | 0.547 | |
| Self-Assessed ConfidenceModel=gemini-2.5-flash2026.02 | 0.53 | 0.544 | |
| SAUPModel=gemini-2.5-pro2026.02 | 0.525 | 0.684 | |
| Normalized EntropyModel=gemini-2.5-flash2026.02 | 0.468 | 0.497 | |
| Self-Assessed ConfidenceModel=gemini-2.5-pro2026.02 | 0.44 | 0.59 | |
| SAUPModel=gemini-2.5-flash2026.02 | 0.428 | 0.467 |