Science Question Answering on ScienceQA v1.0 (test)
82.5Accuracy (G1-4)AUTOACT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| AUTOACTBackbone=Llama-2 70B-chat, Agent Learning Type=Fine-tuning-based, Agent Architecture=Multi-agent2024.01 | 82.5 | 72.5 | 80.83 | 78.61 | |
| ChameleonBackbone=Llama-2 70B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 77.83 | 69.17 | 76.67 | 74.56 | |
| Zero-Shot Plan*Backbone=GPT-3.5 Turbo, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 76.67 | 61.67 | 78.33 | 72.22 | |
| ReflexionBackbone=Llama-2 70B-chat, Agent Learning Type=Fine-tuning-based, Agent Architecture=Single-agent2024.01 | 75.83 | 67.5 | 78.33 | 73.89 | |
| CoTBackbone=Llama-2 70B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 74.17 | 64.17 | 75.83 | 71.39 | |
| FireActBackbone=Llama-2 70B-chat, Agent Learning Type=Fine-tuning-based, Agent Architecture=Single-agent2024.01 | 72.5 | 68.33 | 75 | 71.94 | |
| AUTOACTBackbone=Llama-2 13B-chat, Agent Learning Type=Fine-tuning-based, Agent Architecture=Multi-agent2024.01 | 70.83 | 66.67 | 76.67 | 71.39 | |
| BOLAABackbone=Llama-2 70B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Multi-agent2024.01 | 70 | 67.5 | 75 | 70.83 | |
| AUTOACTBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Fine-tuning-based, Agent Architecture=Multi-agent2024.01 | 69.17 | 68.33 | 72.5 | 70 | |
| ChameleonBackbone=Llama-2 13B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 69.17 | 60.83 | 73.33 | 67.78 | |
| ReflexionBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 67.5 | 65.83 | 69.17 | 67.5 | |
| ReflexionBackbone=Llama-2 13B-chat, Agent Learning Type=Fine-tuning-based, Agent Architecture=Single-agent2024.01 | 67.5 | 64.17 | 73.33 | 68.33 | |
| ChameleonBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 65.83 | 62.5 | 66.67 | 65 | |
| FireActBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Fine-tuning-based, Agent Architecture=Single-agent2024.01 | 65 | 62.5 | 64.17 | 63.89 | |
| ReWOOBackbone=Llama-2 70B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 65 | 61.67 | 76.67 | 67.78 | |
| BOLAABackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Prompt-based, Agent Architecture=Multi-agent2024.01 | 64.17 | 61.67 | 65.83 | 63.89 | |
| ReActBackbone=Llama-2 70B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 64.17 | 60 | 72.5 | 65.56 | |
| ReActBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 63.33 | 58.33 | 62.5 | 61.39 | |
| CoTBackbone=Llama-2 13B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 61.67 | 52.5 | 69.17 | 61.11 | |
| CoTBackbone=GPT-3.5 Turbo, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 60.83 | 55.83 | 65 | 60.56 | |
| ReWOOBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 60.83 | 58.33 | 64.17 | 61.11 | |
| FireActBackbone=Llama-2 13B-chat, Agent Learning Type=Fine-tuning-based, Agent Architecture=Single-agent2024.01 | 60.83 | 57.5 | 67.5 | 61.94 | |
| BOLAABackbone=Llama-2 13B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Multi-agent2024.01 | 60 | 54.17 | 65.83 | 60 | |
| ReActBackbone=Llama-2 13B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 57.5 | 51.67 | 65 | 58.06 | |
| ReWOOBackbone=Llama-2 13B-chat, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 57.5 | 54.17 | 65.83 | 59.17 | |
| CoTBackbone=Mistral-7B Instruct v0.2, Agent Learning Type=Prompt-based, Agent Architecture=Single-agent2024.01 | 54.17 | 50 | 60 | 54.72 |