Roadmap generation on RoadMapper English
66.36StepScoreRoadMapper
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| RoadMapperBase Model=DeepSeek-V3.2, Training=DPO training on Qwen3-32B2026.04 | 66.36 | 72.21 | 85.64 | 92.35 | 79.14 | |
| DyLANBase Model=DeepSeek-V3.22026.04 | 64.37 | 69.96 | 82.2 | 91.47 | 77 | |
| ReConcileBase Model=DeepSeek-V3.22026.04 | 63.44 | 69.82 | 81.12 | 89.77 | 76.04 | |
| RoadMapperBase Model=Qwen3-235B-A22B, Training=DPO training on Qwen3-32B2026.04 | 63.13 | 71.36 | 83.22 | 91.36 | 77.27 | |
| RoadMapperBase Model=Qwen3-14B, Training=DPO training on Qwen3-32B2026.04 | 61.78 | 69.96 | 75.57 | 89.32 | 74.16 | |
| DyLANBase Model=Qwen3-235B-A22B2026.04 | 61.42 | 68.97 | 80.06 | 87.89 | 74.59 | |
| RoadMapperBase Model=Gemini 3 Flash Preview, Training=DPO training on Qwen3-32B2026.04 | 60.47 | 70.27 | 81.73 | 93.07 | 76.39 | |
| ReConcileBase Model=Qwen3-235B-A22B2026.04 | 60.22 | 68.74 | 78.82 | 89.97 | 74.44 | |
| ReConcileBase Model=Gemini 3 Flash Preview2026.04 | 58.62 | 67.91 | 74.89 | 91.33 | 73.19 | |
| CoTBase Model=DeepSeek-V3.22026.04 | 58.46 | 66.87 | 75.68 | 86.71 | 71.93 | |
| RoadMapperBase Model=gpt-oss-20b, Training=DPO training on Qwen3-32B2026.04 | 58.21 | 66.75 | 85.33 | 89.2 | 74.87 | |
| CoTBase Model=Qwen3-235B-A22B2026.04 | 58.2 | 67.96 | 75.46 | 86.53 | 72.04 | |
| Best-of-NBase Model=Qwen3-235B-A22B2026.04 | 57.99 | 67.61 | 75.75 | 86.41 | 71.94 | |
| Best-of-NBase Model=DeepSeek-V3.22026.04 | 57.93 | 66.43 | 74.79 | 87.02 | 71.54 | |
| Direct PromptingBase Model=Qwen3-235B-A22B2026.04 | 57.65 | 67.3 | 74.9 | 86.2 | 71.51 | |
| DyLANBase Model=Gemini 3 Flash Preview2026.04 | 57.29 | 67.37 | 77.54 | 90.94 | 73.29 | |
| Direct PromptingBase Model=DeepSeek-V3.22026.04 | 57.25 | 66.21 | 74.52 | 86.14 | 71.03 | |
| Best-of-NBase Model=gpt-oss-20b2026.04 | 55.74 | 65.89 | 80.65 | 81.99 | 71.07 | |
| CoTBase Model=Gemini 3 Flash Preview2026.04 | 55.11 | 63.64 | 65.7 | 89.72 | 68.54 | |
| Direct PromptingBase Model=gpt-oss-20b2026.04 | 55.04 | 65.47 | 80.05 | 81.41 | 70.49 | |
| Best-of-NBase Model=Qwen3-14B2026.04 | 54.83 | 62.94 | 73.81 | 86.93 | 69.63 | |
| Direct PromptingBase Model=Qwen3-14B2026.04 | 54.76 | 62.64 | 73.56 | 86.04 | 69.25 | |
| Best-of-NBase Model=Gemini 3 Flash Preview2026.04 | 54.7 | 63.48 | 64.81 | 89.95 | 68.24 | |
| RoadMapperBase Model=Llama 4 Maverick, Training=DPO training on Qwen3-32B2026.04 | 54.33 | 65.34 | 69.81 | 92.25 | 70.43 | |
| RoadMapperBase Model=Mistral Small 3.2, Training=DPO training on Qwen3-32B2026.04 | 54.27 | 64.13 | 76.59 | 90.13 | 71.28 | |
| Direct PromptingBase Model=Gemini 3 Flash Preview2026.04 | 54.09 | 63.35 | 64.22 | 89.59 | 67.81 | |
| RoadMapperBase Model=Llama 3.3 70B, Training=DPO training on Qwen3-32B2026.04 | 52.5 | 63.81 | 75.79 | 90.65 | 70.69 | |
| RoadMapperBase Model=Claude 3 Haiku, Training=DPO training on Qwen3-32B2026.04 | 52.33 | 66.74 | 84.3 | 87.37 | 72.69 | |
| RoadMapperBase Model=Llama 3.1 8B, Training=DPO training on Qwen3-32B2026.04 | 49.98 | 62.23 | 71.95 | 86.57 | 67.68 | |
| ReConcileBase Model=Llama 3.3 70B2026.04 | 48.28 | 57.18 | 69.74 | 88.91 | 66.03 | |
| Best-of-NBase Model=Mistral Small 3.22026.04 | 48.1 | 58.41 | 71.52 | 88.11 | 66.54 | |
| DyLANBase Model=Llama 3.3 70B2026.04 | 47.64 | 58.19 | 71.76 | 87.97 | 66.39 | |
| RoadMapperBase Model=GPT-4o mini, Training=DPO training on Qwen3-32B2026.04 | 47.63 | 58.99 | 72.41 | 90.09 | 67.28 | |
| Direct PromptingBase Model=Mistral Small 3.22026.04 | 47.02 | 58.2 | 71.11 | 87.87 | 66.05 | |
| Direct PromptingBase Model=Llama 4 Maverick2026.04 | 46.78 | 56.91 | 61.92 | 86.97 | 63.15 | |
| Best-of-NBase Model=Claude 3 Haiku2026.04 | 46.75 | 59.46 | 80.12 | 79.4 | 66.43 | |
| Best-of-NBase Model=Llama 4 Maverick2026.04 | 46.63 | 56.47 | 61.95 | 87.3 | 63.09 | |
| Direct PromptingBase Model=Claude 3 Haiku2026.04 | 46.12 | 59.28 | 79.84 | 78.35 | 65.9 | |
| Best-of-NBase Model=GPT-4o mini2026.04 | 46.09 | 56.69 | 66.89 | 84.86 | 63.63 | |
| Direct PromptingBase Model=GPT-4o mini2026.04 | 45.73 | 56.02 | 66.31 | 84.86 | 63.23 | |
| CoTBase Model=Llama 3.3 70B2026.04 | 45.39 | 57.1 | 67.21 | 85.83 | 63.88 | |
| Best-of-NBase Model=Llama 3.3 70B2026.04 | 45.15 | 56.49 | 66.97 | 85.87 | 63.62 | |
| Direct PromptingBase Model=Llama 3.3 70B2026.04 | 44.26 | 56.02 | 66.89 | 85.48 | 63.16 | |
| Best-of-NBase Model=Llama 3.1 8B2026.04 | 44.24 | 54.88 | 64.23 | 84.17 | 61.88 | |
| Direct PromptingBase Model=Llama 3.1 8B2026.04 | 42.58 | 54.29 | 64.11 | 83.96 | 61.24 |