Task failure prediction and selective task completion on tau2-bench Telecom 1.0
0.809AUROCTRACER
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| TRACERModel=gemini-2.5-flash2026.02 | 0.809 | 0.52 | |
| TRACERModel=gpt-4.1-mini2026.02 | 0.765 | 0.613 | |
| TRACERModel=gemini-2.5-pro2026.02 | 0.691 | 0.517 | |
| Semantic EntropyModel=gpt-4.1-mini2026.02 | 0.686 | 0.394 | |
| Self-Assessed ConfidenceModel=gpt-4.1-mini2026.02 | 0.683 | 0.389 | |
| SAUPModel=gemini-2.5-flash2026.02 | 0.673 | 0.446 | |
| Semantic EntropyModel=gemini-2.5-pro2026.02 | 0.651 | 0.395 | |
| Normalized EntropyModel=gemini-2.5-pro2026.02 | 0.649 | 0.392 | |
| SAUPModel=gemini-2.5-pro2026.02 | 0.639 | 0.379 | |
| Semantic EntropyModel=gemini-2.5-flash2026.02 | 0.621 | 0.392 | |
| Normalized EntropyModel=gemini-2.5-flash2026.02 | 0.559 | 0.358 | |
| Normalized EntropyModel=gpt-4.1-mini2026.02 | 0.548 | 0.27 | |
| SAUPModel=gpt-4.1-mini2026.02 | 0.534 | 0.347 | |
| Self-Assessed ConfidenceModel=gemini-2.5-pro2026.02 | 0.507 | 0.283 | |
| Self-Assessed ConfidenceModel=gemini-2.5-flash2026.02 | 0.423 | 0.257 |