Dialogue Agent Evaluation on DEMO
7.238Element Awareness - GoalGPT-4o
Evaluation Results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4oCategory=Proprietary LLM, Evaluator=DeepSeek-V32024.12 | 7.238 | 4.272 | 6.646 | 7.86 | 6.504 | 8.183 | 9.146 | 8.565 | 8.832 | 8.682 | 7.23 | |
| Claude-3.5-SonnetCategory=Proprietary LLM, Evaluator=DeepSeek-V32024.12 | 7.09 | 4.46 | 6.772 | 6.534 | 6.213 | 7.734 | 9.141 | 8.404 | 8.815 | 8.523 | 6.979 | |
| Qwen2-72B-InstructCategory=Open-sourced LLM, Evaluator=DeepSeek-V32024.12 | 6.835 | 4.359 | 6.188 | 7.188 | 6.142 | 8.575 | 9.228 | 8.566 | 8.557 | 8.732 | 7.005 | |
| Llama-3.1-70B-InstructCategory=Open-sourced LLM, Evaluator=DeepSeek-V32024.12 | 6.774 | 3.907 | 6.349 | 7.394 | 6.106 | 7.051 | 7.828 | 7.077 | 6.79 | 7.187 | 6.466 | |
| Qwen2-7B-InstructCategory=Backbone LLM, Evaluator=DeepSeek-V32024.12 | 6.655 | 4.025 | 5.582 | 6.925 | 5.797 | 7.161 | 8.557 | 7.304 | 7.376 | 7.6 | 6.398 | |
| DEMO-Llama3.1-8BCategory=DEMO Agent, Evaluator=DeepSeek-V32024.12 | 6.624 | 4.105 | 6.869 | 8.028 | 6.406 | 7.08 | 7.967 | 7.207 | 6.926 | 7.295 | 6.703 | |
| Llama3.1-8B-InstructCategory=Backbone LLM, Evaluator=DeepSeek-V32024.12 | 6.564 | 3.741 | 5.78 | 6.882 | 5.741 | 6.148 | 6.619 | 5.809 | 5.571 | 6.037 | 5.84 | |
| DEMO-Qwen2-7BCategory=DEMO Agent, Evaluator=DeepSeek-V32024.12 | 6.313 | 4.145 | 6.885 | 7.964 | 6.326 | 7.624 | 8.948 | 8.184 | 8.343 | 8.275 | 6.976 | |
| GPT-4o-miniCategory=Proprietary LLM, Evaluator=DeepSeek-V32024.12 | 6.086 | 3.638 | 5.896 | 7.047 | 5.667 | 7.455 | 9.107 | 8.355 | 8.357 | 8.319 | 6.551 | |
| Claude-3.5-HaikuCategory=Proprietary LLM, Evaluator=DeepSeek-V32024.12 | 5.39 | 3.991 | 6.066 | 6.329 | 5.443 | 7.439 | 9.209 | 8.542 | 8.25 | 8.36 | 6.417 |