Program Synthesis on HumanEval (test)
92.37AccuracySC-MAS
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SC-MASLLM backbone=LLM Pool, MAS=true, Routing=true2026.01 | 92.37 | — | — | |
| MasRouterLLM backbone=LLM Pool, MAS=true, Routing=true2026.01 | 90.62 | — | — | |
| AFlowLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.01 | 90.06 | — | — | |
| RouterDCLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 87.75 | — | — | |
| FrugalGPTLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 87.31 | — | — | |
| AgentPruneLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.01 | 86.8 | — | — | |
| VanillaLLM backbone=claude-3.5-haiku, MAS=false, Routing=false2026.01 | 86.33 | — | — | |
| PromptLLMLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 86.33 | — | — | |
| GPTSwarmLLM backbone=gpt-4o-mini, MAS=true, Routing=false2026.01 | 86.28 | — | — | |
| VanillaLLM backbone=gpt-4o-mini, MAS=false, Routing=false2026.01 | 85.71 | — | — | |
| AFlowLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.01 | 85.69 | — | — | |
| RouteLLMLLM backbone=LLM Pool, MAS=false, Routing=true2026.01 | 83.85 | — | — | |
| VanillaLLM backbone=gemini-1.5-flash, MAS=false, Routing=false2026.01 | 82.61 | — | — | |
| AgentPruneLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.01 | 82.55 | — | — | |
| GPTSwarmLLM backbone=gemini-1.5-flash, MAS=true, Routing=false2026.01 | 82.36 | — | — | |
| VanillaLLM backbone=llama-3.1-70b, MAS=false, Routing=false2026.01 | 80.75 | — | — | |
| ConfidenceBackbone=Dream-v0-Instruct-7B, Decoding strategy=Confidence2026.06 | 57.32 | 38.3 | 90.4 | |
| DAWNBackbone=Dream-v0-Instruct-7B, Decoding strategy=DAWN2026.06 | 54.88 | 44.77 | 76.2 | |
| LocalLeapBackbone=Dream-v0-Instruct-7B, Decoding strategy=LocalLeap2026.06 | 54.27 | 42.79 | 76.1 | |
| OriginalBackbone=Dream-v0-Instruct-7B, Decoding strategy=Original2026.06 | 53.66 | 16.62 | 256 | |
| LocalLeap + AXONBackbone=LLaDA-1.5, Decoding strategy=LocalLeap + AXON2026.06 | 44.51 | 20.42 | 77.6 | |
| LocalLeap + AXONCVRBackbone=LLaDA-1.5, Decoding strategy=LocalLeap + AXONCVR2026.06 | 44.51 | 22.91 | 73 | |
| OriginalBackbone=LLaDA-1.5, Decoding strategy=Original2026.06 | 43.9 | 6.04 | 256 | |
| Confidence + AXONBackbone=LLaDA-1.5, Decoding strategy=Confidence + AXON2026.06 | 43.9 | 18.53 | 93.6 | |
| ConfidenceBackbone=LLaDA-1.5, Decoding strategy=Confidence2026.06 | 43.29 | 15.61 | 100.6 | |
| Confidence + AXONCVRBackbone=LLaDA-1.5, Decoding strategy=Confidence + AXONCVR2026.06 | 43.29 | 19.74 | 91 | |
| DAWN + AXONBackbone=LLaDA-1.5, Decoding strategy=DAWN + AXON2026.06 | 42.68 | 26.96 | 62.9 | |
| LocalLeap + AXONBackbone=LLaDA-8B-Instruct, Decoding strategy=LocalLeap + AXON2026.06 | 42.68 | 76.66 | 63.1 | |
| DAWNBackbone=LLaDA-1.5, Decoding strategy=DAWN2026.06 | 42.07 | 22.96 | 77.3 | |
| LocalLeap + AXONCVRBackbone=LLaDA-8B-Instruct, Decoding strategy=LocalLeap + AXONCVR2026.06 | 42.06 | 79.39 | 60.2 | |
| LocalLeapBackbone=LLaDA-1.5, Decoding strategy=LocalLeap2026.06 | 41.46 | 19.83 | 75.2 | |
| ConfidenceBackbone=LLaDA-8B-Instruct, Decoding strategy=Confidence2026.06 | 40.85 | 58.96 | 77.9 | |
| Confidence + AXONBackbone=LLaDA-8B-Instruct, Decoding strategy=Confidence + AXON2026.06 | 40.55 | 65.45 | 74.6 | |
| Confidence + AXONCVRBackbone=LLaDA-8B-Instruct, Decoding strategy=Confidence + AXONCVR2026.06 | 40.55 | 66.23 | 73.6 | |
| OriginalBackbone=LLaDA-8B-Instruct, Decoding strategy=Original2026.06 | 40.24 | 18.09 | 256 | |
| DAWNBackbone=LLaDA-8B-Instruct, Decoding strategy=DAWN2026.06 | 40.24 | 80.26 | 63.2 | |
| DAWN + AXONBackbone=LLaDA-8B-Instruct, Decoding strategy=DAWN + AXON2026.06 | 39.98 | 92 | 53.2 | |
| LocalLeapBackbone=LLaDA-8B-Instruct, Decoding strategy=LocalLeap2026.06 | 39.63 | 71.2 | 61.4 | |
| DAWN + AXONCVRBackbone=LLaDA-8B-Instruct, Decoding strategy=DAWN + AXONCVR2026.06 | 39.58 | 92.22 | 52.4 | |
| DAWN + AXONCVRBackbone=LLaDA-1.5, Decoding strategy=DAWN + AXONCVR2026.06 | 39.02 | 29.11 | 59.6 |