Travel Planning on TravelPlanner (test)
87.4Final Pass RateQwen3-14B
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Qwen3-14BStrategy=FBOS-RL2026.05 | 87.4 | 98.5 | 91.8 | 96.07 | 92.9 | — | |
| Qwen3-14BStrategy=GRPO2026.05 | 84.2 | 97.83 | 89.5 | 94.28 | 91.4 | — | |
| Llama-3.1-8B-InstructStrategy=FBOS-RL2026.05 | 68.7 | 98.81 | 94 | 84.11 | 71.7 | — | |
| Llama-3.1-8B-InstructStrategy=GRPO2026.05 | 65.2 | 98.83 | 91.4 | 84.54 | 69.7 | — | |
| Qwen3-8BTraining=CARL2026.07 | 56.1 | 97.3 | 81.7 | 73.1 | 59.4 | — | |
| Qwen3-8BTraining=RFT2026.07 | 45 | 96.3 | 74.4 | 65.7 | 48.9 | — | |
| DeepSeek-R1Strategy=CoT2026.05 | 40 | 92.65 | 55.2 | 74.15 | 62 | — | |
| DICE-PCPlanning Setting=Sole-planning, Backbone=GPT-42026.06 | 15.2 | — | — | — | — | 100 | |
| DICE-FTPlanning Setting=Sole-planning, Backbone=Qwen3-32B2026.06 | 12.7 | — | — | — | — | 100 | |
| DeepSeek-R12026.07 | 12.2 | 80.6 | 22.2 | 51.7 | 41.7 | — | |
| DeepSeek-R1-Distill-Llama-8BTraining=CARL2026.07 | 11.7 | 81.1 | 32.1 | 42.9 | 28.9 | — | |
| MA-DebatePlanning Setting=Sole-planning, Backbone=Qwen3-235B2026.06 | 11.3 | — | — | — | — | 99.8 | |
| DICE-PCPlanning Setting=Sole-planning, Backbone=Qwen3-32B2026.06 | 10.3 | — | — | — | — | 100 | |
| o1-preview2026.07 | 10 | 79.6 | 15 | 41.9 | 37.8 | — | |
| MA-FTPlanning Setting=Sole-planning, Backbone=Qwen3-32B2026.06 | 9.7 | — | — | — | — | 100 | |
| DICEPlanning Setting=Two-stage, Backbone=GPT-42026.06 | 9.3 | — | — | — | — | 100 | |
| Qwen3-8BStrategy=prompt reflect2026.05 | 8.3 | 73.84 | 14.6 | 27.95 | 24 | — | |
| GPT-4o2026.07 | 7.8 | 84.7 | 31.1 | 53.6 | 31.1 | — | |
| DICE-FTPlanning Setting=Two-stage, Backbone=Qwen3-32B2026.06 | 7.1 | — | — | — | — | 100 | |
| DebatePlanning Setting=Sole-planning, Backbone=GPT-4, Rounds=32026.06 | 7.1 | — | — | — | — | 98.2 | |
| GPT-4o-2024-11-20Strategy=CoT2026.05 | 7 | 84.18 | 25.9 | 49.87 | 26.6 | — | |
| Qwen2.5-72B-Instruct2026.07 | 6.1 | 82.3 | 16.7 | 32.6 | 22.8 | — | |
| Qwen3-8BStrategy=CoT2026.05 | 5.9 | 72.04 | 10.7 | 28 | 21.8 | — | |
| DirectPlanning Setting=Sole-planning, Backbone=GPT-o32026.06 | 5.8 | — | — | — | — | 100 | |
| MA-DebatePlanning Setting=Two-stage, Backbone=Qwen3-235B2026.06 | 5.7 | — | — | — | — | 98.9 | |
| DeepSeek-R1-Distill-Llama-8BTraining=RFT2026.07 | 5.6 | 80.8 | 25 | 36.2 | 19.4 | — | |
| DICE-PCPlanning Setting=Two-stage, Backbone=Qwen3-32B2026.06 | 5.4 | — | — | — | — | 99.1 | |
| GPT-4-TurboStrategy=Direct2026.05 | 4.4 | 80.6 | 15.2 | 44.3 | 23.1 | — | |
| DirectPlanning Setting=Sole-planning, Backbone=GPT-4-Turbo2026.06 | 4.4 | — | — | — | — | 100 | |
| QwQ-32B2026.07 | 4.4 | 74.9 | 6.1 | 41.4 | 32.8 | — | |
| MA-FTPlanning Setting=Two-stage, Backbone=Qwen3-32B2026.06 | 4.3 | — | — | — | — | 98.6 | |
| DebatePlanning Setting=Two-stage, Backbone=GPT-4, Rounds=32026.06 | 3.7 | — | — | — | — | 97.8 | |
| Qwen3-14B2026.05 | 2.9 | 66.62 | 6.13 | 19.03 | 15.5 | — | |
| DeepSeek-V32026.07 | 2.2 | 80.3 | 17.2 | 30.5 | 13.9 | — | |
| Llama-3.1-70B-Instruct2026.07 | 2.2 | 82.8 | 18.9 | 33.1 | 16.1 | — | |
| Qwen3-8B2026.07 | 2.2 | 72.7 | 7.8 | 34.8 | 27.8 | — | |
| Gemini ProStrategy=Direct2026.05 | 2.1 | 64.7 | 7.9 | 10.6 | 4.7 | — | |
| GPT-o3Planning Setting=Two-stage2026.06 | 1.9 | — | — | — | — | 95.6 | |
| GPT-3.5-TurboStrategy=ReAct2026.05 | 0.7 | 45.9 | 2.5 | 10.7 | 3.1 | — | |
| Mixtral-8x7B-MoEStrategy=Direct2026.05 | 0.7 | 67 | 3.7 | 3.9 | 1.6 | — | |
| GPT-3.5-TurboStrategy=Direct2026.05 | 0.6 | 59.5 | 2.7 | 9.5 | 4.4 | — | |
| GPT-3.5-TurboStrategy=Reflexion2026.05 | 0.6 | 52.1 | 2.2 | 9.9 | 3.8 | — | |
| GPT-4-TurboPlanning Setting=Two-stage2026.06 | 0.6 | — | — | — | — | 93.1 | |
| Llama-3.1-8B-Instruct2026.05 | 0.43 | 57.12 | 1.13 | 6.75 | 3.53 | — | |
| GPT-3.5-TurboStrategy=CoT2026.05 | 0.4 | 64.4 | 2.3 | 9.8 | 3.8 | — | |
| Greedy Search2026.05 | 0 | 72 | 0 | 52.4 | 31.8 | — | |
| Greedy Search2026.06 | 0 | — | — | — | — | 100 | |
| Llama-3.1-8B-Instruct2026.07 | 0 | 60.1 | 0 | 7.9 | 2.8 | — | |
| DeepSeek-R1-Distill-Llama-8B2026.07 | 0 | 61.2 | 0 | 0 | 0 | — |