Multi-turn Text-to-SQL on CoSQL & SParC Aggregate
66.6Avg Exact Match (EM)[Short-Horizon] SFT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| [Short-Horizon] SFTModel Size=4B, Horizon Setting=Short-Horizon2025.10 | 66.6 | 74.1 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome + Process)Model Size=4B, Horizon Setting=Long-Horizon2025.10 | 66.5 | 77.6 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome Only)Model Size=4B, Horizon Setting=Long-Horizon2025.10 | 66.1 | 76.9 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome + Process)Model Size=1.7B, Horizon Setting=Long-Horizon2025.10 | 64.4 | 74.6 | |
| [Short-Horizon] Direct RLModel Size=4B, Horizon Setting=Short-Horizon2025.10 | 64.2 | 74 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome Only)Model Size=1.7B, Horizon Setting=Long-Horizon2025.10 | 63.7 | 74 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 3)Model Size=4B, Horizon Setting=Long-Horizon, Learning Round=Round 32025.10 | 63.7 | 74.2 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 2)Model Size=4B, Horizon Setting=Long-Horizon, Learning Round=Round 22025.10 | 62.8 | 74.2 | |
| [Short-Horizon] SFTModel Size=1.7B, Horizon Setting=Short-Horizon2025.10 | 62.2 | 69.6 | |
| [Long-Horizon] Warm-Start SFT Only (Round 1)Model Size=4B, Horizon Setting=Long-Horizon, Learning Round=Round 12025.10 | 62 | 73.6 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 3)Model Size=1.7B, Horizon Setting=Long-Horizon, Learning Round=Round 32025.10 | 61.7 | 71.5 | |
| [Short-Horizon] Direct RLModel Size=1.7B, Horizon Setting=Short-Horizon2025.10 | 60.7 | 70.5 | |
| HIE-SQL+GraPPModel Size=~125M2025.10 | 60.6 | — | |
| [Long-Horizon] + Warm-Start SFT Only (Round 2)Model Size=1.7B, Horizon Setting=Long-Horizon, Learning Round=Round 22025.10 | 59.8 | 70.4 | |
| [Long-Horizon] Warm-Start SFT Only (Round 1)Model Size=1.7B, Horizon Setting=Long-Horizon, Learning Round=Round 12025.10 | 58.3 | 68.7 | |
| RASAT+PICARDModel Size=3B2025.10 | 57.7 | 64.5 | |
| Qwen-3-32BModel Size=32B2025.10 | 53.9 | 70.4 | |
| Qwen-3-14BModel Size=14B2025.10 | 53.1 | 65.3 | |
| CoE-SQLModel Size=Closed-Source, Prompting Strategy=Few-shot, 16-shot2025.10 | 51.6 | 64.1 | |
| Qwen-3-4BModel Size=4B2025.10 | 50.3 | 63.5 | |
| ACT-SQLModel Size=Closed-Source2025.10 | 48.5 | 63.8 | |
| Qwen-3-1.7BModel Size=1.7B2025.10 | 47.9 | 60.7 | |
| GAZP+BERTModel Size=~215M2025.10 | 45.5 | 43.3 | |
| [Long-Horizon] Base Agent without TrainingModel Size=4B, Horizon Setting=Long-Horizon2025.10 | 44.9 | 59 | |
| DeepSeek-R1Model Size=671B2025.10 | 36.6 | 58.1 | |
| Planning/Tool-using Agent built by LangGraphModel Size=Closed-Source2025.10 | 33.6 | 69.8 | |
| GPT-4.1Model Size=Closed-Source2025.10 | 32.7 | 61.4 | |
| OpenAI-O3Model Size=Closed-Source2025.10 | 29.7 | 58.4 | |
| [Long-Horizon] Base Agent without TrainingModel Size=1.7B, Horizon Setting=Long-Horizon2025.10 | 17.1 | 23.3 |