Dissatisfied-turn Detection on human turn-level satisfaction annotations
36.55F1 ScoreUser-memory evaluator
Evaluation Results
| Method | Links | |
|---|---|---|
| User-memory evaluatorType=User-aware LLM judge, Backbone=Qwen3-8B2026.05 | 36.55 | |
| SPUR-style evaluatorType=Generic LLM judge, Backbone=Qwen3-8B2026.05 | 29.55 | |
| Few-shot judgeType=Generic LLM judge, Backbone=Qwen3-8B2026.05 | 26.8 | |
| Task-rubric judgeType=Generic LLM judge, Backbone=Qwen3-8B2026.05 | 25.59 | |
| Nearest-history scoreType=User-history retrieval2026.05 | 23.61 | |
| BERT scorerType=Supervised scorer, Backbone=BERT2026.05 | 22.68 | |
| Prometheus-rubric judgeType=Generic LLM judge, Backbone=Qwen3-8B2026.05 | 22.07 | |
| BERT ordinal scorerType=Supervised scorer, Backbone=BERT2026.05 | 19.84 | |
| Zero-shot judgeType=Generic LLM judge, Backbone=Qwen3-8B2026.05 | 17.32 | |
| RAG prompted scorerType=User-history retrieval2026.05 | 2.14 |