Text-to-SQL on BIRD-SQL Mini (dev)
63.8Execution Accuracy (EX)PV-SQL
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PV-SQLInput tokens=4270, Output tokens=315, Average task completion time (seconds)=4.12026.04 | 63.8 | — | — | — | — | — | — | — | — | |
| PV-SQL (w/o Repair)Input tokens=4261, Output tokens=276, Average task completion time (seconds)=4.32026.04 | 60.8 | — | — | — | — | — | — | — | — | |
| PV-SQL (w/o Probe)Input tokens=1191, Output tokens=81, Average task completion time (seconds)=1.42026.04 | 60.6 | — | — | — | — | — | — | — | — | |
| TA-SQLBase Model=GPT-4.1-mini2026.04 | 60.2 | — | — | — | — | — | — | — | 66.11 | |
| PV-SQLBase Model=GPT-4.1-mini2026.04 | 60.2 | — | — | — | — | — | — | — | 67.15 | |
| TA-SQLInput tokens=6365, Output tokens=240, Average task completion time (seconds)=6.72026.04 | 58.4 | — | — | — | — | — | — | — | — | |
| MAC-SQLInput tokens=6203, Output tokens=380, Average task completion time (seconds)=4.72026.04 | 57.8 | — | — | — | — | — | — | — | — | |
| E-SQLInput tokens=29986, Output tokens=708, Average task completion time (seconds)=24.42026.04 | 57.4 | — | — | — | — | — | — | — | — | |
| MAC-SQLBase Model=GPT-4.1-mini2026.04 | 55.6 | — | — | — | — | — | — | — | 60.62 | |
| TS-SQLInput tokens=1891, Output tokens=362, Average task completion time (seconds)=4.12026.04 | 55 | — | — | — | — | — | — | — | — | |
| E-SQLBase Model=GPT-4.1-mini2026.04 | 54.8 | — | — | — | — | — | — | — | 60.95 | |
| DIN-SQLBase Model=GPT-4.1-mini2026.04 | 53.8 | — | — | — | — | — | — | — | 58.22 | |
| PV-SQL (LLM verify)Input tokens=4983, Output tokens=508, Average task completion time (seconds)=6.72026.04 | 53.8 | — | — | — | — | — | — | — | — | |
| XiYan-SQLInput tokens=1734, Output tokens=142, Average task completion time (seconds)=1.72026.04 | 52.2 | — | — | — | — | — | — | — | — | |
| XiYan-SQLBase Model=GPT-4.1-mini2026.04 | 51.6 | — | — | — | — | — | — | — | 58.99 | |
| DIN-SQLInput tokens=26630, Output tokens=694, Average task completion time (seconds)=25.02026.04 | 51 | — | — | — | — | — | — | — | — | |
| DAIL-SQLBase Model=GPT-4.1-mini2026.04 | 50.4 | — | — | — | — | — | — | — | 55.05 | |
| TS-SQLBase Model=GPT-4.1-mini2026.04 | 50.2 | — | — | — | — | — | — | — | 54.87 | |
| DAIL-SQLInput tokens=2247, Output tokens=60, Average task completion time (seconds)=0.92026.04 | 50 | — | — | — | — | — | — | — | — | |
| GPT-4.1-mini2026.04 | 49.4 | — | — | — | — | — | — | — | 54.86 | |
| ZS-CoTInput tokens=1927, Output tokens=211, Average task completion time (seconds)=2.12026.04 | 49.4 | — | — | — | — | — | — | — | — | |
| Claude Sonnet 4.5Action policy=Qwen3-4B-Instruct2026.03 | — | 70.8 | 63.8 | 67.8 | 57.3 | 63 | 64.5 | — | — | |
| GEPAAction policy=Qwen3-4B-Instruct2026.03 | — | 64 | 62 | 72 | 54 | 62 | 62.8 | — | — | |
| GPT-5Action policy=Qwen3-4B-Instruct2026.03 | — | 70.8 | 65.8 | 72 | 54.3 | 63.3 | 65.2 | — | — | |
| GPT-5Shot count=0-shot2025.11 | — | — | — | — | — | — | 29 | 5,942 | — | |
| GPT-5Shot count=N-shot2025.11 | — | — | — | — | — | — | 48 | 6,674 | — | |
| LSEAction policy=Qwen3-4B-Instruct2026.03 | — | 72 | 68.5 | 72 | 59.8 | 64 | 67.3 | — | — | |
| MultiGA (Ensemble GA)Eval Model=Qwen3, Shot count=3-shot2025.11 | — | — | — | — | — | — | 70.5 | 157,430 | — | |
| MultiGA (Ensemble GB)Eval Model=GPT-5, Shot count=3-shot2025.11 | — | — | — | — | — | — | 68.5 | 134,930 | — | |
| MultiGA (Gemma-2 seed)Eval Model=Qwen3, Shot count=3-shot2025.11 | — | — | — | — | — | — | 47.5 | 68,651 | — | |
| MultiGA (GPT-5 seed)Eval Model=Qwen3, Shot count=3-shot2025.11 | — | — | — | — | — | — | 58.5 | 133,085 | — | |
| MultiGA (Llama-4 seed)Eval Model=Qwen3, Shot count=3-shot2025.11 | — | — | — | — | — | — | 63 | 117,428 | — | |
| MultiGA (Phi-4-Mini seed)Eval Model=Qwen3, Shot count=3-shot2025.11 | — | — | — | — | — | — | 16.5 | 232,313 | — | |
| Qwen3Shot count=0-shot2025.11 | — | — | — | — | — | — | 48 | 4,624 | — | |
| Qwen3Shot count=N-shot2025.11 | — | — | — | — | — | — | 49 | 5,585 | — | |
| Qwen3-4B-InstructAction policy=Qwen3-4B-Instruct2026.03 | — | 63.7 | 60.3 | 70.2 | 56 | 61 | 62.2 | — | — | |
| Seed promptAction policy=Qwen3-4B-Instruct2026.03 | — | 51 | 60.3 | 63.7 | 54.5 | 56.5 | 57.2 | — | — | |
| TextGradAction policy=Qwen3-4B-Instruct2026.03 | — | 60.3 | 66 | 71.5 | 56.5 | 61.3 | 63.1 | — | — |