Graduate-level Science Question Answering on GPQA (Accuracy & Loss)
61.95Accuracy (GPQA)Zero-Shot CoT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Zero-Shot CoTCategory=Open-Source 30B+ Model, Model=QwenR1 32B2025.03 | 61.95 | — | |
| ARM + MPFoundation Model=GPT-4.1-nano2025.10 | 61.4 | — | |
| ARMFoundation Model=GPT-4.1-nano2025.10 | 60.1 | — | |
| ARM + MPFoundation Model=GPT-4o2025.10 | 60.1 | — | |
| ARMFoundation Model=GPT-4o2025.10 | 59.5 | — | |
| SKILL-MOECategory=Multi-Model Multi-Agent, Model=Adaptive2025.03 | 57.78 | — | |
| LLM-DebateFoundation Model=GPT-4o2025.10 | 56.3 | — | |
| SIGMAMulti-agent collaboration=full support, Inter-agent relations=full support, Handling conflicting signals=full support2026.05 | 54.51 | — | |
| CoTFoundation Model=GPT-4o2025.10 | 53.8 | — | |
| Self-RefineFoundation Model=GPT-4o2025.10 | 53.8 | — | |
| AFlowFoundation Model=GPT-4o, Evaluation Setup=test set2025.10 | 53.8 | — | |
| Self-Consistency (SC)Category=Advanced Single Model, Model=Task-Best x52025.03 | 53.54 | — | |
| CoT-SCFoundation Model=GPT-4o2025.10 | 53.2 | — | |
| CoT-SCFoundation Model=LLaMA-3.3-70B2025.10 | 53.2 | — | |
| GoAMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 53.16 | — | |
| Self-MoACategory=Single-Model Multi-Agent, Model=Task-Best x32025.03 | 52.86 | — | |
| MoACategory=Multi-Model Multi-Agent, Model=Top-32025.03 | 52.86 | — | |
| Self-MoAMulti-agent collaboration=full support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 52.73 | — | |
| G-DesignerMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 52.72 | — | |
| MoAMulti-agent collaboration=full support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 52.54 | — | |
| LLM-DebateFoundation Model=GPT-4.1-nano2025.10 | 52.5 | — | |
| Zero-Shot CoTCategory=Open-Source 30B+ Model, Model=Llama3.3 70B2025.03 | 51.44 | — | |
| AFlowFoundation Model=GPT-4.1-nano, Evaluation Setup=1000-sample2025.10 | 51.3 | — | |
| Self-RefineFoundation Model=LLaMA-3.3-70B2025.10 | 51.3 | — | |
| Zero-Shot CoTCategory=Open-Source 30B+ Model, Model=Qwen2.5 72B2025.03 | 51.02 | — | |
| GPTSwarmMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 51.01 | — | |
| Self-Refine (SR)Category=Advanced Single Model, Model=Task-Best2025.03 | 50.84 | — | |
| CoT-SCFoundation Model=GPT-4.1-nano2025.10 | 50.6 | — | |
| AFlowFoundation Model=GPT-4o, Evaluation Setup=1000-sample2025.10 | 50.6 | — | |
| LLM-DebateFoundation Model=LLaMA-3.3-70B2025.10 | 50.6 | — | |
| Multi-Agent DebateCategory=Single-Model Multi-Agent, Model=Task-Best x32025.03 | 50.51 | — | |
| AutoGenMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 50.45 | — | |
| Complete GraphMulti-agent collaboration=full support, Inter-agent relations=full support, Handling conflicting signals=no support2026.05 | 50.41 | — | |
| CoTFoundation Model=GPT-4.1-nano2025.10 | 50 | — | |
| Self-RefineFoundation Model=GPT-4.1-nano2025.10 | 50 | — | |
| CoTFoundation Model=LLaMA-3.3-70B2025.10 | 50 | — | |
| ARM + MPFoundation Model=LLaMA-3.3-70B2025.10 | 50 | — | |
| DyLANMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 49.72 | — | |
| ARMFoundation Model=LLaMA-3.3-70B2025.10 | 49.6 | — | |
| Random GraphMulti-agent collaboration=full support, Inter-agent relations=full support, Handling conflicting signals=no support2026.05 | 49.33 | — | |
| Zero-Shot CoTCategory=Open-Source 7B Model, Model=Task-Best2025.03 | 48.43 | — | |
| PHPMulti-agent collaboration=full support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 48.32 | — | |
| ADASFoundation Model=GPT-4.1-nano, Evaluation Setup=test set2025.10 | 48.1 | — | |
| ReConcileCategory=Multi-Model Multi-Agent, Model=Top-32025.03 | 47.98 | — | |
| ADASFoundation Model=LLaMA-3.3-70B, Evaluation Setup=test set2025.10 | 47.5 | — | |
| ComplexCoTMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 47.48 | — | |
| CoTMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 47.21 | — | |
| SCMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 47.2 | — | |
| ADASFoundation Model=GPT-4.1-nano, Evaluation Setup=1000-sample2025.10 | 46.8 | — | |
| ADASFoundation Model=GPT-4o, Evaluation Setup=1000-sample2025.10 | 46.8 | — | |
| AFlowFoundation Model=LLaMA-3.3-70B, Evaluation Setup=test set2025.10 | 46.8 | — | |
| AFlowFoundation Model=LLaMA-3.3-70B, Evaluation Setup=1000-sample2025.10 | 46.8 | — | |
| ADASFoundation Model=GPT-4o, Evaluation Setup=test set2025.10 | 46.2 | — | |
| VanillaMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 45.96 | — | |
| Zero-Shot CoTCategory=Open-Source 7B Model, Model=QwenR1 7B2025.03 | 44.95 | — | |
| ADASFoundation Model=LLaMA-3.3-70B, Evaluation Setup=1000-sample2025.10 | 42.4 | — | |
| AFlowFoundation Model=GPT-4.1-nano, Evaluation Setup=test set2025.10 | 39.9 | — | |
| I-DPO + MaPPOBase Model=Qwen2.5-7B-Instruct2026.05 | 33.3 | — | |
| DPO + MaPPOBase Model=Qwen2.5-7B-Instruct2026.05 | 33.1 | — | |
| I-DPOBase Model=Qwen2.5-7B-Instruct2026.05 | 33 | — | |
| AdamWModel Size=3B, Optimizer=AdamW2026.04 | 32.8 | 2.005 | |
| DPOBase Model=Qwen2.5-7B-Instruct2026.05 | 32 | — | |
| ITBase Model=Qwen2.5-7B-Instruct2026.05 | 31.5 | — | |
| NexusModel Size=1B, Optimizer=Nexus2026.04 | 30.4 | 2.201 | |
| NexusModel Size=3B, Optimizer=Nexus2026.04 | 29.6 | 1.981 | |
| Frozen checkpointModel=DeepSeek-MoE-Chat2026.05 | 28.91 | — | |
| ST-MoEModel=DeepSeek-MoE-Chat2026.05 | 28.75 | — | |
| phi-balancingModel=DeepSeek-MoE-Chat2026.05 | 27.88 | — | |
| phi-balancingModel=DeepSeek-V2-Lite2026.05 | 26.21 | — | |
| ST-MoEModel=DeepSeek-V2-Lite2026.05 | 25.48 | — | |
| AdamWModel Size=1B, Optimizer=AdamW2026.04 | 25 | 2.221 | |
| Frozen checkpointModel=DeepSeek-V2-Lite2026.05 | 19.98 | — |