End-to-End Selection on AlgoSimBench MCQ
75.1AccuracyASM-PL
Evaluation Results
| Method | Links | |
|---|---|---|
| ASM-PLModel=o3-mini-medium2025.07 | 75.1 | |
| ASM-NLModel=o3-mini-medium2025.07 | 74.4 | |
| Solution*Model=o3-mini-medium2025.07 | 72.6 | |
| Solution*Model=Deepseek-R12025.07 | 70.6 | |
| Solution*Model=Claude-3.5-Sonnet2025.07 | 70.5 | |
| ASM-PLModel=Deepseek-R12025.07 | 70.4 | |
| Solution*Model=Gemini-2.0-Flash2025.07 | 69.7 | |
| ASM-NLModel=Deepseek-R12025.07 | 69.2 | |
| ASM-NLModel=Qwen3-Coder-480B2025.07 | 68.4 | |
| Solution*Model=Qwen3-Coder-480B2025.07 | 66.9 | |
| Solution*Model=Deepseek-V32025.07 | 66.7 | |
| StatementModel=o3-mini-medium2025.07 | 65.9 | |
| ASM-NLModel=Deepseek-V32025.07 | 64.9 | |
| StatementModel=Deepseek-R12025.07 | 63.7 | |
| Solution*Model=GPT-4o2025.07 | 63.4 | |
| SummaryModel=o3-mini-medium2025.07 | 63.4 | |
| ASM-PLModel=Deepseek-V32025.07 | 62.7 | |
| Solution*Model=Qwen3-8B-thinking2025.07 | 62.4 | |
| ASM-PLModel=Qwen3-Coder-480B2025.07 | 62.4 | |
| Solution*Model=Qwen3-4B-thinking2025.07 | 59.2 | |
| ASM-NLModel=Qwen3-8B-thinking2025.07 | 58.2 | |
| ASM-NLModel=Gemini-2.0-Flash2025.07 | 57.9 | |
| SummaryModel=Deepseek-R12025.07 | 57.7 | |
| ASM-PLModel=Qwen3-8B-thinking2025.07 | 57 | |
| ASM-PLModel=Gemini-2.0-Flash2025.07 | 55.5 | |
| StatementModel=Deepseek-V32025.07 | 55.2 | |
| ASM-NLModel=Claude-3.5-Sonnet2025.07 | 54.7 | |
| Solution*Model=GPT-4o-mini2025.07 | 54.4 | |
| ASM-PLModel=Qwen3-4B-thinking2025.07 | 53.9 | |
| ASM-NLModel=GPT-4o2025.07 | 53.2 | |
| SummaryModel=Deepseek-V32025.07 | 53.2 | |
| ASM-NLModel=Qwen3-4B-thinking2025.07 | 53 | |
| ASM-PLModel=GPT-4o2025.07 | 53 | |
| ASM-PLModel=Claude-3.5-Sonnet2025.07 | 53 | |
| StatementModel=Gemini-2.0-Flash2025.07 | 51.2 | |
| SummaryModel=Qwen3-Coder-480B2025.07 | 51 | |
| StatementModel=Qwen3-Coder-480B2025.07 | 50.7 | |
| SummaryModel=Gemini-2.0-Flash2025.07 | 48.5 | |
| StatementModel=Qwen3-8B-thinking2025.07 | 47.5 | |
| SummaryModel=Qwen3-8B-thinking2025.07 | 46.3 | |
| SummaryModel=Claude-3.5-Sonnet2025.07 | 45 | |
| StatementModel=Claude-3.5-Sonnet2025.07 | 44.2 | |
| ASM-NLModel=GPT-4o-mini2025.07 | 43.8 | |
| StatementModel=Qwen3-4B-thinking2025.07 | 43.3 | |
| ASM-PLModel=GPT-4o-mini2025.07 | 42.5 | |
| StatementModel=GPT-4o2025.07 | 41.5 | |
| SummaryModel=Qwen3-4B-thinking2025.07 | 41.2 | |
| SummaryModel=GPT-4o2025.07 | 38.1 | |
| SummaryModel=GPT-4o-mini2025.07 | 35.8 | |
| StatementModel=GPT-4o-mini2025.07 | 35.6 | |
| ASM-NLModel=Qwen3-0.6B-thinking2025.07 | 33.3 | |
| ASM-PLModel=Qwen3-0.6B-thinking2025.07 | 31.3 | |
| Solution*Model=Qwen3-0.6B-thinking2025.07 | 30.6 | |
| Solution*Model=Llama3.1-8B-Instruct2025.07 | 29.6 | |
| ASM-NLModel=Llama3.1-8B-Instruct2025.07 | 26.4 | |
| SummaryModel=Qwen3-0.6B-thinking2025.07 | 26.1 | |
| StatementModel=Qwen3-0.6B-thinking2025.07 | 25.8 | |
| ASM-PLModel=Llama3.1-8B-Instruct2025.07 | 24.4 | |
| SummaryModel=Llama3.1-8B-Instruct2025.07 | 24.1 | |
| StatementModel=Llama3.1-8B-Instruct2025.07 | 23.9 |