Element Awareness on DEMO
6.508Goal Recognition ScoreGPT-4o
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| GPT-4oModel Category=Proprietary LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.508 | 5.781 | 6.452 | 7.616 | 6.602 | — | |
| GPT-4o-miniModel Category=Proprietary LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.386 | 5.519 | 6.21 | 7.294 | 6.372 | — | |
| Llama-3.1-70B-InstructModel Category=Open-sourced LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.359 | 5.64 | 6.509 | 7.32 | 6.472 | — | |
| Qwen2-72B-InstructModel Category=Open-sourced LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.292 | 5.882 | 6.168 | 7.087 | 6.365 | — | |
| DEMO-Llama3.1-8BModel Category=DEMO Agent, Evaluator=Gemini-1.5-Pro2024.12 | 6.264 | 5.759 | 6.661 | 8.107 | 6.717 | — | |
| Claude-3.5-SonnetModel Category=Proprietary LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.195 | 5.507 | 6.281 | 6.567 | 6.148 | — | |
| Qwen2-7B-InstructModel Category=Backbone LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.154 | 5.403 | 5.869 | 6.684 | 6.041 | — | |
| Llama3.1-8B-InstructModel Category=Backbone LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.112 | 5.161 | 5.95 | 7.025 | 6.077 | — | |
| Claude-3.5-HaikuModel Category=Proprietary LLM, Evaluator=Gemini-1.5-Pro2024.12 | 6.1 | 5.741 | 6.632 | 6.645 | 6.292 | — | |
| DEMO-Qwen2-7BModel Category=DEMO Agent, Evaluator=Gemini-1.5-Pro2024.12 | 6.037 | 5.683 | 6.61 | 8.108 | 6.629 | — | |
| Claude-3.5-SonnetModel Category=Proprietary LLM, Temperature=02024.12 | 5.979 | 4.145 | 6.221 | 6.243 | 5.647 | 6.599 | |
| GPT-4oModel Category=Proprietary LLM, Temperature=02024.12 | 5.975 | 4.051 | 6.167 | 7.308 | 5.875 | 6.793 | |
| GPT-4o-miniModel Category=Proprietary LLM, Temperature=02024.12 | 5.802 | 3.586 | 5.748 | 7.002 | 5.534 | 6.449 | |
| DEMO-Llama3.1-8BModel Category=DEMO Agent, Backbone=Llama3.1-8B-Instruct, Temperature=02024.12 | 5.623 | 3.939 | 6.543 | 7.926 | 6.008 | 6.341 | |
| Llama-3.1-70B-InstructModel Category=Open-sourced LLM, Temperature=02024.12 | 5.559 | 3.643 | 6.078 | 7.051 | 5.593 | 6.154 | |
| Llama3.1-8B-InstructModel Category=Backbone LLM, Temperature=02024.12 | 5.546 | 3.287 | 5.403 | 6.523 | 5.189 | 5.335 | |
| Claude-3.5-HaikuModel Category=Proprietary LLM, Temperature=02024.12 | 5.492 | 3.858 | 6.071 | 6.304 | 5.431 | 6.358 | |
| Qwen2-72B-InstructModel Category=Open-sourced LLM, Temperature=02024.12 | 5.357 | 4.406 | 5.702 | 6.921 | 5.596 | 6.608 | |
| Qwen2-7B-InstructModel Category=Backbone LLM, Temperature=02024.12 | 5.306 | 3.981 | 5.459 | 6.347 | 5.244 | 5.828 | |
| DEMO-Qwen2-7BModel Category=DEMO Agent, Backbone=Qwen2-7B-Instruct, Temperature=02024.12 | 5.229 | 3.946 | 6.534 | 7.914 | 5.906 | 6.625 |