Human Agreement Evaluation on Interactive Agent Evaluation Criteria
0.7Average Agreement ScoreOnline Agent-as-a-Judge
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Online Agent-as-a-Judge2026.06 | 0.7 | 0.58 | 0.68 | 0.33 | 0.76 | 0.82 | 0.72 | 0.73 | 0.96 | |
| Offline Agent-as-a-Judge2026.06 | 0.4 | 0.47 | 0.38 | 0.14 | 0.52 | 0.55 | 0.44 | 0.33 | 0.17 | |
| Offline LLM-as-a-Judge2026.06 | 0.33 | 0.42 | 0.38 | 0 | 0.26 | 0.45 | 0.44 | 0.33 | 0.17 |