Query Generation for Agent Failure Discovery on Shopper-Agent Interactions 100 products (test)
4.51Realism ScoreHuman
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HumanRefinement Strategy=Baseline, Evaluator Model=gpt-5-mini2026.05 | 4.51 | — | — | 0.53 | 0 | 45 | 78 | 88 | 0 | 88.9 | 0 | |
| GEPARefinement Strategy=Prompt Refinement, Evaluator Model=gpt-5-mini2026.05 | 3.81 | 45.6 | 1.3 | 0.36 | -32.08 | 33 | 68 | 75 | -18.09 | 50.33 | -43.39 | |
| PQRRefinement Strategy=Query Refinement, Evaluator Model=gpt-5-mini2026.05 | 3.66 | 56.32 | 1 | 0.55 | 3.77 | 36 | 77 | 88 | -7.09 | 92.4 | 3.94 | |
| ACERefinement Strategy=Prompt Refinement, Evaluator Model=gpt-5-mini2026.05 | 2.66 | 39.58 | 1.1 | 0.43 | -18.87 | 38 | 75 | 85 | -7.6 | 80.68 | -9.25 | |
| TAPRefinement Strategy=Query Refinement, Evaluator Model=gpt-5-mini2026.05 | 2.31 | 33.38 | 1.6 | 0.56 | 5.66 | 42 | 83 | 96 | 2.94 | 197.02 | 121.62 | |
| PAIRRefinement Strategy=Query Refinement, Evaluator Model=gpt-5-mini2026.05 | 1.83 | 31.57 | 2.8 | 0.58 | 9.43 | 34 | 77 | 91 | -7.44 | 159.95 | 79.92 |