False Success Detection on tau2-bench v1 v2 (test)
89RecallTF-IDF + XGBoost
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TF-IDF + XGBoostFlag rate=20%2026.06 | 89 | 31 | — | |
| TF-IDF + XGBoostFlag rate=10%2026.06 | 72 | 50 | — | |
| TF-IDF + XGBoostFlag rate=5%2026.06 | 51 | 72 | — | |
| Judge (Sonnet 4.5)Flag rate=20%, Condition=no closing2026.06 | 25 | 55 | — | |
| Judge (Sonnet 4.5)Flag rate=10%, Condition=no closing2026.06 | 13 | 58 | — | |
| Judge (Sonnet 4.5)Flag rate=5%, Condition=no closing2026.06 | 6 | 56 | — | |
| Judge (Sonnet 4.5)Interface=LLM API call2026.06 | — | — | 4,000 | |
| TF-IDF + XGBoostHardware=CPU2026.06 | — | — | 1.19 |