Multi-turn Text-to-SQL on SParC In-domain
71.5Exact Match[Short-Horizon] SFT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| [Short-Horizon] SFTModel Size=4B, Horizon Setting=Short-Horizon2025.10 | 71.5 | 78.3 | |
| [Short-Horizon] SFTModel Size=1.7B, Horizon Setting=Short-Horizon2025.10 | 69.2 | 74.3 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome + Process)Model Size=4B, Horizon Setting=Long-Horizon2025.10 | 68.7 | 79 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome Only)Model Size=4B, Horizon Setting=Long-Horizon2025.10 | 67.8 | 78.1 | |
| RASAT+PICARDModel Size=3B2025.10 | 67.7 | 73.3 | |
| [Short-Horizon] Direct RLModel Size=4B, Horizon Setting=Short-Horizon2025.10 | 66.5 | 75.8 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome Only)Model Size=1.7B, Horizon Setting=Long-Horizon2025.10 | 66.1 | 76.2 | |
| [Long-Horizon] Warm-Start SFT + RL (Outcome + Process)Model Size=1.7B, Horizon Setting=Long-Horizon2025.10 | 66.1 | 76.2 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 3)Model Size=1.7B, Horizon Setting=Long-Horizon, Learning Round=Round 32025.10 | 65.7 | 72.8 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 3)Model Size=4B, Horizon Setting=Long-Horizon, Learning Round=Round 32025.10 | 65.6 | 75.1 | |
| [Short-Horizon] Direct RLModel Size=1.7B, Horizon Setting=Short-Horizon2025.10 | 65.5 | 72.1 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 2)Model Size=4B, Horizon Setting=Long-Horizon, Learning Round=Round 22025.10 | 64.8 | 74.9 | |
| HIE-SQL+GraPPModel Size=~125M2025.10 | 64.7 | — | |
| [Long-Horizon] Warm-Start SFT Only (Round 1)Model Size=4B, Horizon Setting=Long-Horizon, Learning Round=Round 12025.10 | 63.1 | 73.8 | |
| [Long-Horizon] + Warm-Start SFT Only (Round 2)Model Size=1.7B, Horizon Setting=Long-Horizon, Learning Round=Round 22025.10 | 63 | 72.3 | |
| [Long-Horizon] Warm-Start SFT Only (Round 1)Model Size=1.7B, Horizon Setting=Long-Horizon, Learning Round=Round 12025.10 | 62 | 70.6 | |
| CoE-SQLModel Size=Closed-Source, Prompting Strategy=Few-shot, 16-shot2025.10 | 56 | 70.3 | |
| Qwen-3-32BModel Size=32B2025.10 | 53.4 | 74 | |
| Qwen-3-14BModel Size=14B2025.10 | 51.9 | 64.1 | |
| ACT-SQLModel Size=Closed-Source2025.10 | 51 | 63.8 | |
| Qwen-3-4BModel Size=4B2025.10 | 49.8 | 62.9 | |
| GAZP+BERTModel Size=~215M2025.10 | 48.9 | 47.8 | |
| Qwen-3-1.7BModel Size=1.7B2025.10 | 46.5 | 61.5 | |
| [Long-Horizon] Base Agent without TrainingModel Size=4B, Horizon Setting=Long-Horizon2025.10 | 44.1 | 57.6 | |
| DeepSeek-R1Model Size=671B2025.10 | 37.2 | 57.6 | |
| Planning/Tool-using Agent built by LangGraphModel Size=Closed-Source2025.10 | 34.6 | 69.6 | |
| GPT-4.1Model Size=Closed-Source2025.10 | 33.3 | 61.8 | |
| OpenAI-O3Model Size=Closed-Source2025.10 | 30.3 | 57 | |
| [Long-Horizon] Base Agent without TrainingModel Size=1.7B, Horizon Setting=Long-Horizon2025.10 | 17.8 | 23.9 |