Text-to-SQL on BIRD (EX, VES)
65.12Execution Accuracy (EX)PV-SQL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| PV-SQLBase LLM=GPT-4o2026.04 | 65.12 | 75.55 | |
| PV-SQLBase LLM=GPT-4o2026.04 | 65.12 | 75.55 | |
| PV-SQLBase LLM=GPT-4.1-mini2026.04 | 63.62 | 86.9 | |
| TA-SQLBase LLM=GPT-4o2026.04 | 60.43 | 62.99 | |
| TA-SQLBase LLM=GPT-4o2026.04 | 60.43 | 62.99 | |
| TA-SQLBase LLM=GPT-4.1-mini2026.04 | 60.2 | 66.11 | |
| PV-SQLBase LLM=GPT-OSS-20B2026.04 | 59.45 | 61.99 | |
| E-SQLBase LLM=GPT-4o2026.04 | 59.1 | 64.65 | |
| E-SQLBase LLM=GPT-4o2026.04 | 59.1 | 64.65 | |
| MAC-SQLBase LLM=GPT-4o2026.04 | 58.7 | 62.77 | |
| MAC-SQLBase LLM=GPT-4o2026.04 | 58.7 | 62.77 | |
| XiYan-SQLBase LLM=GPT-4o2026.04 | 57.6 | 63.96 | |
| XiYan-SQLBase LLM=GPT-4o2026.04 | 57.6 | 63.96 | |
| TS-SQLBase LLM=GPT-4o2026.04 | 57.37 | 63.43 | |
| TS-SQLBase LLM=GPT-4o2026.04 | 57.37 | 63.43 | |
| TA-SQLBase LLM=GPT-OSS-20B2026.04 | 55.74 | 55.51 | |
| MAC-SQLBase LLM=GPT-4.1-mini2026.04 | 55.6 | 60.62 | |
| DIN-SQLBase LLM=GPT-4o2026.04 | 54.8 | 56.75 | |
| DIN-SQLBase LLM=GPT-4o2026.04 | 54.8 | 56.75 | |
| E-SQLBase LLM=GPT-4.1-mini2026.04 | 54.8 | 60.95 | |
| DAIL-SQLBase LLM=GPT-4o2026.04 | 54.1 | 58.77 | |
| DAIL-SQLBase LLM=GPT-4o2026.04 | 54.1 | 58.77 | |
| E-SQLBase LLM=GPT-OSS-20B2026.04 | 53.95 | 53.05 | |
| DIN-SQLBase LLM=GPT-4.1-mini2026.04 | 53.8 | 58.22 | |
| MAC-SQLBase LLM=GPT-OSS-20B2026.04 | 53.59 | 51.5 | |
| TS-SQLBase LLM=GPT-4.1-mini2026.04 | 53.52 | 58.81 | |
| CoTE-SQLCategory=Fine-tuned-based methods, Model=Qwen2.5-7B2026.06 | 53.39 | 59.02 | |
| GPT-4o2026.04 | 52.22 | 56.99 | |
| ZS-CoTBase LLM=GPT-4o2026.04 | 52.22 | 56.99 | |
| XiYan-SQLBase LLM=GPT-4.1-mini2026.04 | 51.6 | 58.99 | |
| TS-SQLBase LLM=GPT-OSS-20B2026.04 | 50.98 | 55.2 | |
| ZS-CoTBase LLM=GPT-OSS-20B2026.04 | 50.85 | 52.84 | |
| DAIL-SQLBase LLM=GPT-4.1-mini2026.04 | 50.4 | 55.05 | |
| ZS-CoTBase LLM=GPT-4.1-mini2026.04 | 50.26 | 53.75 | |
| DIN-SQLBase LLM=GPT-OSS-20B2026.04 | 50.03 | 46.56 | |
| PV-SQLBase LLM=Qwen3-4B2026.04 | 49.61 | 51.47 | |
| CoTE-SQLCategory=Fine-tuned-based methods, Model=Llama-3.1-8B2026.06 | 49.54 | 55.25 | |
| DAIL-SQLBase LLM=GPT-OSS-20B2026.04 | 49.39 | 48.22 | |
| DACCategory=Shallow prompt-based methods, Model=Qwen2.5-7B2026.06 | 46.54 | 48.25 | |
| GPT-4Category=Direct, Model=GPT-42026.06 | 46.35 | 49.77 | |
| DACCategory=Shallow prompt-based methods, Model=Llama-3.1-8B2026.06 | 45.57 | 48.32 | |
| Qwen2.5-7BCategory=Direct, Model=Qwen2.5-7B2026.06 | 45.44 | 48.37 | |
| MAC-SQLCategory=Fine-tuned-based methods, Model=SQL-Llama (7B)2026.06 | 43.94 | 57.36 | |
| DTS-SQLCategory=Fine-tuned-based methods, Model=Qwen2.5-7B2026.06 | 42.18 | 44.43 | |
| DTS-SQLCategory=Fine-tuned-based methods, Model=Llama-3.1-8B2026.06 | 41.92 | 43.59 | |
| E-SQLBase LLM=Qwen3-4B2026.04 | 41.66 | 46.38 | |
| Llama-3.1-8BCategory=Direct, Model=Llama-3.1-8B2026.06 | 41.2 | 46.58 | |
| DAIL-SQLCategory=Shallow prompt-based methods, Model=Llama-3.1-8B2026.06 | 41 | 43.82 | |
| DIN-SQLCategory=CoT-based methods, Model=Qwen2.5-7B2026.06 | 39.96 | 41.41 | |
| DAIL-SQLCategory=Shallow prompt-based methods, Model=Qwen2.5-7B2026.06 | 39.9 | 41.59 | |
| MAC-SQLBase LLM=Qwen3-4B2026.04 | 39.76 | 40.88 | |
| PV-SQLBase LLM=Gemma3-4B2026.04 | 39.5 | 42.39 | |
| DIN-SQLCategory=CoT-based methods, Model=Llama-3.1-8B2026.06 | 38.72 | 40.65 | |
| MAC-SQLBase LLM=Gemma3-4B2026.04 | 37.7 | 37.36 | |
| ZS-CoTBase LLM=Qwen3-4B2026.04 | 36.44 | 39.01 | |
| XiYan-SQLBase LLM=Qwen3-4B2026.04 | 36.4 | 38.31 | |
| PURPLECategory=Shallow prompt-based methods, Model=Qwen2.5-7B2026.06 | 36.31 | 42.33 | |
| XiYan-SQLBase LLM=GPT-OSS-20B2026.04 | 35.98 | 38.03 | |
| ZS-CoTBase LLM=Gemma3-4B2026.04 | 34.55 | 35.65 | |
| TA-SQLBase LLM=Qwen3-4B2026.04 | 30.2 | 32.48 | |
| DAIL-SQLBase LLM=Qwen3-4B2026.04 | 28.81 | 29.99 | |
| PURPLECategory=Shallow prompt-based methods, Model=Llama-3.1-8B2026.06 | 28.49 | — | |
| TS-SQLBase LLM=Qwen3-4B2026.04 | 27.4 | 31.47 | |
| DAIL-SQLBase LLM=Gemma3-4B2026.04 | 27.32 | 27.41 | |
| E-SQLBase LLM=Gemma3-4B2026.04 | 26.2 | 29.14 | |
| TS-SQLBase LLM=Gemma3-4B2026.04 | 25.02 | 25.29 | |
| DIN-SQLBase LLM=Qwen3-4B2026.04 | 23.44 | 22.51 | |
| TA-SQLBase LLM=Gemma3-4B2026.04 | 23.01 | 22.71 | |
| DIN-SQLBase LLM=Gemma3-4B2026.04 | 22.23 | 20.57 | |
| XiYan-SQLBase LLM=Gemma3-4B2026.04 | 20.79 | 30.69 | |
| DeepSeek-R1-Llama-8BCategory=Direct, Model=DeepSeek-R1-Llama-8B2026.06 | 15.12 | 15.98 | |
| DEA-SQLCategory=CoT-based methods, Model=Qwen2.5-7B2026.06 | 13.17 | 14.93 | |
| ZS-CoTBase LLM=Qwen3-0.6B2026.04 | 13.1 | 15.52 | |
| E-SQLBase LLM=Qwen3-0.6B2026.04 | 12.84 | 13.06 | |
| MAC-SQLBase LLM=Qwen3-0.6B2026.04 | 11.6 | 12.95 | |
| PV-SQLBase LLM=Qwen3-0.6B2026.04 | 11.08 | 15.39 | |
| DEA-SQLCategory=CoT-based methods, Model=Llama-3.1-8B2026.06 | 8.41 | 8.35 | |
| DAIL-SQLBase LLM=Qwen3-0.6B2026.04 | 8.15 | 8.5 | |
| DIN-SQLBase LLM=Qwen3-0.6B2026.04 | 6.84 | 7.13 | |
| XiYan-SQLBase LLM=Qwen3-0.6B2026.04 | 6.6 | 6.93 | |
| DeepSeek-R1-Qwen-7BCategory=Direct, Model=DeepSeek-R1-Qwen-7B2026.06 | 6.17 | 7.05 | |
| TS-SQLBase LLM=Qwen3-0.6B2026.04 | 5.6 | 8.48 | |
| TA-SQLBase LLM=Qwen3-0.6B2026.04 | 3 | 3.16 |