List your company

Task failure prediction and selective task completion on tau2-bench Retail 1.0

0.707AUROCTRACER
0.416840.492170.56750.64283Feb 11, 2026

Evaluation results

MethodLinks
TRACERModel=gemini-2.5-flash2026.020.7070.67
TRACERModel=gpt-4.1-mini2026.020.6890.632
TRACERModel=gemini-2.5-pro2026.020.6730.725
Semantic EntropyModel=gpt-4.1-mini2026.020.620.579
Normalized EntropyModel=gpt-4.1-mini2026.020.6170.571
SAUPModel=gpt-4.1-mini2026.020.6090.584
Self-Assessed ConfidenceModel=gpt-4.1-mini2026.020.5770.555
Semantic EntropyModel=gemini-2.5-pro2026.020.5560.673
Normalized EntropyModel=gemini-2.5-pro2026.020.5530.661
Semantic EntropyModel=gemini-2.5-flash2026.020.5330.547
Self-Assessed ConfidenceModel=gemini-2.5-flash2026.020.530.544
SAUPModel=gemini-2.5-pro2026.020.5250.684
Normalized EntropyModel=gemini-2.5-flash2026.020.4680.497
Self-Assessed ConfidenceModel=gemini-2.5-pro2026.020.440.59
SAUPModel=gemini-2.5-flash2026.020.4280.467