Roadmap generation on RoadMapper Chinese
67.62StepScoreRoadMapper
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| RoadMapperBase Model=DeepSeek-V3.2, Training=DPO training on Qwen3-32B2026.04 | 67.62 | 72.46 | 87.32 | 92.93 | 80.08 | |
| RoadMapperBase Model=Qwen3-235B-A22B, Training=DPO training on Qwen3-32B2026.04 | 66.46 | 71.94 | 88.36 | 93.82 | 80.15 | |
| ReConcileBase Model=DeepSeek-V3.22026.04 | 65.98 | 69.7 | 83.56 | 90.54 | 77.45 | |
| RoadMapperBase Model=Gemini 3 Flash Preview, Training=DPO training on Qwen3-32B2026.04 | 64.91 | 72.34 | 81.88 | 93.64 | 78.19 | |
| RoadMapperBase Model=Qwen3-14B, Training=DPO training on Qwen3-32B2026.04 | 64.74 | 71.7 | 75.99 | 91.61 | 76.01 | |
| DyLANBase Model=DeepSeek-V3.22026.04 | 64.55 | 69.41 | 86.84 | 90.04 | 77.71 | |
| ReConcileBase Model=Qwen3-235B-A22B2026.04 | 63.04 | 69.52 | 84.69 | 91.79 | 77.26 | |
| DyLANBase Model=Qwen3-235B-A22B2026.04 | 61.81 | 70.53 | 85.24 | 91.22 | 77.2 | |
| ReConcileBase Model=Gemini 3 Flash Preview2026.04 | 61.22 | 68.5 | 76.08 | 90.82 | 74.16 | |
| Best-of-NBase Model=Qwen3-235B-A22B2026.04 | 60.87 | 67.84 | 80.17 | 90.57 | 74.86 | |
| CoTBase Model=DeepSeek-V3.22026.04 | 60.83 | 67.53 | 77.86 | 88.08 | 73.58 | |
| CoTBase Model=Qwen3-235B-A22B2026.04 | 60.76 | 67.61 | 80.37 | 90.84 | 74.9 | |
| Best-of-NBase Model=DeepSeek-V3.22026.04 | 60.47 | 67.38 | 77.72 | 88.92 | 73.62 | |
| Direct PromptingBase Model=Qwen3-235B-A22B2026.04 | 60.35 | 66.62 | 79.66 | 90.57 | 74.3 | |
| Direct PromptingBase Model=DeepSeek-V3.22026.04 | 59.85 | 66.95 | 77.59 | 88.42 | 73.2 | |
| RoadMapperBase Model=gpt-oss-20b, Training=DPO training on Qwen3-32B2026.04 | 59.6 | 66.88 | 87.29 | 88.81 | 75.65 | |
| DyLANBase Model=Gemini 3 Flash Preview2026.04 | 59.6 | 68.81 | 74.49 | 91.28 | 73.55 | |
| CoTBase Model=Gemini 3 Flash Preview2026.04 | 57.64 | 66.68 | 69.23 | 91.24 | 71.2 | |
| Best-of-NBase Model=Gemini 3 Flash Preview2026.04 | 57.19 | 66.23 | 69.21 | 90.79 | 70.86 | |
| Direct PromptingBase Model=Gemini 3 Flash Preview2026.04 | 57.12 | 65.89 | 68.67 | 90.63 | 70.58 | |
| Best-of-NBase Model=Qwen3-14B2026.04 | 56.64 | 64.19 | 74.88 | 90.01 | 71.43 | |
| Direct PromptingBase Model=Qwen3-14B2026.04 | 56.39 | 64.11 | 74.06 | 90.13 | 71.17 | |
| RoadMapperBase Model=Mistral Small 3.2, Training=DPO training on Qwen3-32B2026.04 | 55.12 | 63 | 83.95 | 91.47 | 73.39 | |
| Best-of-NBase Model=gpt-oss-20b2026.04 | 53.83 | 60.59 | 81.37 | 81.97 | 69.44 | |
| RoadMapperBase Model=Llama 4 Maverick, Training=DPO training on Qwen3-32B2026.04 | 53.11 | 64.35 | 74.32 | 93.3 | 71.27 | |
| Direct PromptingBase Model=gpt-oss-20b2026.04 | 53.05 | 60.5 | 80.88 | 81.97 | 69.1 | |
| RoadMapperBase Model=Claude 3 Haiku, Training=DPO training on Qwen3-32B2026.04 | 50.89 | 64.35 | 84.33 | 90.11 | 72.42 | |
| RoadMapperBase Model=Llama 3.3 70B, Training=DPO training on Qwen3-32B2026.04 | 50.05 | 60.93 | 78.94 | 90.89 | 70.2 | |
| Direct PromptingBase Model=Mistral Small 3.22026.04 | 48.58 | 59.91 | 74.34 | 86.18 | 67.25 | |
| RoadMapperBase Model=Llama 3.1 8B, Training=DPO training on Qwen3-32B2026.04 | 48.51 | 58.79 | 76.36 | 88.15 | 67.95 | |
| Best-of-NBase Model=Mistral Small 3.22026.04 | 48.35 | 59.57 | 74.34 | 86.32 | 67.15 | |
| DyLANBase Model=Llama 3.3 70B2026.04 | 46.86 | 56.7 | 75.24 | 87.32 | 66.53 | |
| Best-of-NBase Model=Claude 3 Haiku2026.04 | 46.35 | 58.97 | 81.36 | 82.97 | 67.41 | |
| Best-of-NBase Model=Llama 4 Maverick2026.04 | 46.29 | 56.39 | 64.44 | 88.59 | 63.93 | |
| Direct PromptingBase Model=Llama 4 Maverick2026.04 | 46.14 | 55.61 | 64.37 | 88.36 | 63.62 | |
| RoadMapperBase Model=GPT-4o mini, Training=DPO training on Qwen3-32B2026.04 | 46.03 | 57 | 74.57 | 90.74 | 67.09 | |
| Direct PromptingBase Model=Claude 3 Haiku2026.04 | 45.97 | 58.64 | 80.37 | 82.9 | 66.97 | |
| ReConcileBase Model=Llama 3.3 70B2026.04 | 45.45 | 57.21 | 75.89 | 87.34 | 66.47 | |
| Best-of-NBase Model=GPT-4o mini2026.04 | 44.16 | 53.45 | 68.27 | 86.5 | 63.1 | |
| Direct PromptingBase Model=GPT-4o mini2026.04 | 43.97 | 53.05 | 67.92 | 85.73 | 62.67 | |
| Best-of-NBase Model=Llama 3.3 70B2026.04 | 40.24 | 51.16 | 71.5 | 84.55 | 61.86 | |
| Best-of-NBase Model=Llama 3.1 8B2026.04 | 39.75 | 48.97 | 70.94 | 83.66 | 60.83 | |
| CoTBase Model=Llama 3.3 70B2026.04 | 39.66 | 51.19 | 71.72 | 84.93 | 61.88 | |
| Direct PromptingBase Model=Llama 3.3 70B2026.04 | 38.16 | 50.94 | 70.74 | 84.3 | 61.04 | |
| Direct PromptingBase Model=Llama 3.1 8B2026.04 | 37.63 | 48.81 | 70.23 | 83.37 | 60.01 |