Text-to-SQL on BIRD
97.59Total Execution AccuracyGemma 3
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 3Best alpha=0.35, Strategy=Best Pairwise2026.05 | 97.59 | — | — | — | — | — | — | — | — | |
| Llama 3.1Best alpha=0.35, Strategy=Best Pairwise2026.05 | 96.67 | — | — | — | — | — | — | — | — | |
| SQCoderBest alpha=0.38, Strategy=Best Pairwise2026.05 | 95.29 | — | — | — | — | — | — | — | — | |
| Qwen 3Best alpha=0.26, Strategy=Best Pairwise2026.05 | 88.63 | — | — | — | — | — | — | — | — | |
| IQuest-Coder-V1-40B-InstructModel Size Category=20B+2026.03 | 70.5 | — | — | — | — | — | — | — | — | |
| IQuest-Coder-V1-40B-Loop-InstructModel Size Category=20B+2026.03 | 69.9 | — | — | — | — | — | — | — | — | |
| Qwen-PlusModel Scale Group=Larger Parameter & Proprietary Models2025.12 | 68.32 | — | — | — | — | — | — | — | — | |
| Gemini-3-Pro-previewModel Size Category=Closed-APIs Models2026.03 | 67.5 | — | — | — | — | — | — | — | — | |
| Gemini-3-Flash-previewModel Size Category=Closed-APIs Models2026.03 | 66.6 | — | — | — | — | — | — | — | — | |
| DCTREmbedding model=stella, k=52026.03 | 66.56 | — | — | — | — | — | — | — | — | |
| BaselineContext=Full schema, Model=Gemini2026.03 | 66.3 | — | — | — | — | — | — | — | — | |
| DCTREmbedding model=stella, k=102026.03 | 66.3 | — | — | — | — | — | — | — | — | |
| DCTREmbedding model=stella, k=252026.03 | 66.23 | — | — | — | — | — | — | — | — | |
| Claude-Opus-4.5Model Size Category=Closed-APIs Models2026.03 | 66 | — | — | — | — | — | — | — | — | |
| DeepSeek-V3Model Scale Group=Larger Parameter & Proprietary Models2025.12 | 65.65 | — | — | — | — | — | — | — | — | |
| GPT-4oModel Scale Group=Larger Parameter & Proprietary Models2025.12 | 65.25 | — | — | — | — | — | — | — | — | |
| RSL-SQLEvidence Setting=with BIRD evidence, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 64.67 | 71.68 | 55.6 | 48.97 | — | — | — | — | — | |
| TableGPT-R1-8BModel Scale Group=Comparable Parameter Models2025.12 | 63.17 | — | — | — | — | — | — | — | — | |
| Qwen3-32BModel Scale Group=Larger Parameter & Proprietary Models2025.12 | 63.04 | — | — | — | — | — | — | — | — | |
| Qwen3-235B-A22B-Instruct-2507Model Size Category=20B+2026.03 | 62.8 | — | — | — | — | — | — | — | — | |
| Claude-Sonnet-4.5Model Size Category=Closed-APIs Models2026.03 | 62.5 | — | — | — | — | — | — | — | — | |
| CHESSEvidence Setting=with BIRD evidence, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 62.14 | 67.38 | 56.82 | 47.06 | — | — | — | — | — | |
| Qwen2.5-Coder-32B-InstructModel Size Category=20B+2026.03 | 62.1 | — | — | — | — | — | — | — | — | |
| Qwen3-14BModel Scale Group=Larger Parameter & Proprietary Models2025.12 | 61.8 | — | — | — | — | — | — | — | — | |
| Qwen3-8BModel Scale Group=Comparable Parameter Models2025.12 | 61.67 | — | — | — | — | — | — | — | — | |
| Qwen3-Coder-480B-A35B-InstructModel Size Category=20B+2026.03 | 61.3 | — | — | — | — | — | — | — | — | |
| Kimi-K2-Instruct-0905Model Size Category=20B+2026.03 | 60.4 | — | — | — | — | — | — | — | — | |
| MAC-SQLLanguage Model=GPT-42026.02 | 59.59 | — | — | — | — | — | — | 67.68 | — | |
| Qwen2.5-Coder-14B-InstructModel Size Category=13B+2026.03 | 59.1 | — | — | — | — | — | — | — | — | |
| Qwen3-Coder-30B-A3B-InstructModel Size Category=13B+2026.03 | 59 | — | — | — | — | — | — | — | — | |
| DAIL-SQLEvidence Setting=with BIRD evidence, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 57.89 | 63.78 | 51.72 | 40 | — | — | — | — | — | |
| DAIL-SQLLanguage Model=GPT-42026.02 | 57.41 | — | — | — | — | — | — | 61.95 | — | |
| RSL-SQL (CA)Evidence Setting=Companion Agents, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 56.45 | 65.19 | 44.4 | 39.31 | — | — | — | — | — | |
| DIN-SQLLanguage Model=GPT-42026.02 | 55.9 | — | — | — | — | — | — | 59.44 | — | |
| Table-SpecialistBase Model=GPT-4, Training Strategy=Table-Specialist2024.10 | 55.6 | — | — | — | — | — | — | — | — | |
| IQuest-Coder-V1-40B-Loop-ThinkingModel Size Category=20B+2026.03 | 54.8 | — | — | — | — | — | — | — | — | |
| QwQ-32BModel Scale Group=Larger Parameter & Proprietary Models2025.12 | 54.3 | — | — | — | — | — | — | — | — | |
| Qwen3-72BModel Scale Group=Larger Parameter & Proprietary Models2025.12 | 53.91 | — | — | — | — | — | — | — | — | |
| DAIL-SQL (CA)Evidence Setting=Companion Agents, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 53.65 | 62.16 | 42.67 | 34.48 | — | — | — | — | — | |
| IQuest-Coder-V1-40B-ThinkingModel Size Category=20B+2026.03 | 53.6 | — | — | — | — | — | — | — | — | |
| GPT-5.1Model Size Category=Closed-APIs Models2026.03 | 53.3 | — | — | — | — | — | — | — | — | |
| Qwen2.5-Coder-7B-InstructModel Size Category=6B+2026.03 | 53.1 | — | — | — | — | — | — | — | — | |
| GPT-4Base Model=GPT-4, Training Strategy=Vanilla2024.10 | 52.8 | — | — | — | — | — | — | — | — | |
| DeepSeek-v3.2Model Size Category=20B+2026.03 | 52.6 | — | — | — | — | — | — | — | — | |
| KAT-DevModel Size Category=20B+2026.03 | 52.2 | — | — | — | — | — | — | — | — | |
| CHESS (CA)Evidence Setting=Companion Agents, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 52.15 | 59.46 | 43.1 | 34.48 | — | — | — | — | — | |
| RSL-SQLEvidence Setting=without BIRD evidence, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 51.96 | 61.41 | 40.09 | 29.66 | — | — | — | — | — | |
| Table-SpecialistBase Model=GPT-3.5, Training Strategy=Table-Specialist2024.10 | 51.4 | — | — | — | — | — | — | — | — | |
| Table-R1-Zero-7BModel Scale Group=Comparable Parameter Models2025.12 | 50.98 | — | — | — | — | — | — | — | — | |
| IQuest-Coder-V1-14B-InstructModel Size Category=13B+2026.03 | 50 | — | — | — | — | — | — | — | — | |
| TableGPT2-7BModel Scale Group=Comparable Parameter Models2025.12 | 49.28 | — | — | — | — | — | — | — | — | |
| CHESSEvidence Setting=without BIRD evidence, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 47.78 | 56 | 37.93 | 26.9 | — | — | — | — | — | |
| Proposed AgenticLanguage Model=Mistral, Llama, GPT-4o, Fine Tuned=No2026.02 | 47.78 | — | — | — | — | — | — | 51.05 | — | |
| IQuest-Coder-V1-14B-ThinkingModel Size Category=13B+2026.03 | 46.9 | — | — | — | — | — | — | — | — | |
| GLM-4.7Model Size Category=20B+2026.03 | 46.5 | — | — | — | — | — | — | — | — | |
| GPT-3.5Base Model=GPT-3.5, Training Strategy=Vanilla2024.10 | 45.2 | — | — | — | — | — | — | — | — | |
| Seed-Coder-8B-InstructModel Size Category=6B+2026.03 | 44.7 | — | — | — | — | — | — | — | — | |
| Table-SpecialistBase Model=Llama3.1-8B, Training Strategy=Table-Specialist2024.10 | 43.8 | — | — | — | — | — | — | — | — | |
| DeepSeek-Coder-V2-Lite-InstructModel Size Category=6B+2026.03 | 41.6 | — | — | — | — | — | — | — | — | |
| Kimi-K2-ThinkingModel Size Category=20B+2026.03 | 40.6 | — | — | — | — | — | — | — | — | |
| Llama-3.1-8BModel Scale Group=Comparable Parameter Models2025.12 | 40.03 | — | — | — | — | — | — | — | — | |
| DAIL-SQLEvidence Setting=without BIRD evidence, temperature=0.2, top-p=0.9, max_tokens=1024, few-shot retrieval k=52025.12 | 39.52 | 38.76 | 27.39 | 17.77 | — | — | — | — | — | |
| Llama3.1-8BBase Model=Llama3.1-8B, Training Strategy=Vanilla2024.10 | 38.8 | — | — | — | — | — | — | — | — | |
| IQuest-Coder-V1-7B-InstructModel Size Category=6B+2026.03 | 37.7 | — | — | — | — | — | — | — | — | |
| Qwen3-235B-A22B-Thinking-2507Model Size Category=20B+2026.03 | 35.2 | — | — | — | — | — | — | — | — | |
| KAT-Dev-72B-ExpModel Size Category=20B+2026.03 | 35.2 | — | — | — | — | — | — | — | — | |
| TableLLMModel Scale Group=Comparable Parameter Models2025.12 | 30.64 | — | — | — | — | — | — | — | — | |
| IQuest-Coder-V1-7B-ThinkingModel Size Category=6B+2026.03 | 30.5 | — | — | — | — | — | — | — | — | |
| DAPOBackbone=Llama-3-8B2026.03 | — | — | — | — | 60.1 | — | 69.2 | — | — | |
| DATAMINDBackbone=Qwen-2.5-Coder-7B2025.09 | — | — | — | — | 59.41 | 69.88 | — | — | — | |
| DATAMINDBackbone=Qwen-2.5-Coder-14B2025.09 | — | — | — | — | 62.23 | 70.21 | — | — | — | |
| DPH-RLBackbone=Llama-3-8B2026.03 | — | — | — | — | 62.8 | — | 72.4 | — | — | |
| DyJRBackbone=Llama-3-8B2026.03 | — | — | — | — | 62.7 | — | 72.9 | — | — | |
| GPT-OSS-120B2026.04 | — | — | — | — | — | — | — | — | 38.25 | |
| GRPOBackbone=Llama-3-8B2026.03 | — | — | — | — | 59.4 | — | 68.4 | — | — | |
| Nemotron 3 Super2026.04 | — | — | — | — | — | — | — | — | 41.8 | |
| OmniSQLBackbone=Qwen-2.5-Coder-7B2025.09 | — | — | — | — | 57.11 | 66.3 | — | — | — | |
| OmniSQLBackbone=Qwen-2.5-Coder-14B2025.09 | — | — | — | — | 58.8 | 67.41 | — | — | — | |
| ReActBackbone=GPT-4o2025.09 | — | — | — | — | 50.2 | 62.39 | — | — | — | |
| ReActBackbone=o4-mini2025.09 | — | — | — | — | 57.04 | 66.88 | — | — | — | |
| ReActBackbone=DeepSeek-R12025.09 | — | — | — | — | 55.8 | 66.17 | — | — | — | |
| ReActBackbone=DeepSeek-V3.12025.09 | — | — | — | — | 57.89 | 68.12 | — | — | — | |
| ReActBackbone=GPT-52025.09 | — | — | — | — | 60.17 | 65.19 | — | — | — | |
| ReActBackbone=Qwen-2.5-Coder-32B2025.09 | — | — | — | — | 41.2 | 60.17 | — | — | — | |
| ReActBackbone=QwQ-32B2025.09 | — | — | — | — | 50.3 | 64.21 | — | — | — | |
| ReActBackbone=Llama-3.3-70B2025.09 | — | — | — | — | 59.1 | 68.58 | — | — | — | |
| ReActBackbone=Qwen-2.5-72B2025.09 | — | — | — | — | 60.3 | 69.49 | — | — | — | |
| ReActBackbone=Qwen-2.5-Coder-7B2025.09 | — | — | — | — | 7.02 | 18.71 | — | — | — | |
| ReActBackbone=Qwen-2.5-Coder-14B2025.09 | — | — | — | — | 41.76 | 59.91 | — | — | — | |
| SQL-R1Backbone=Qwen-2.5-Coder-7B2025.09 | — | — | — | — | 56.78 | 66.23 | — | — | — | |
| SQL-R1Backbone=Qwen-2.5-Coder-14B2025.09 | — | — | — | — | 58.02 | 66.62 | — | — | — | |
| Table-R1Backbone=Qwen-2.5-Coder-7B2025.09 | — | — | — | — | 10.69 | 13.49 | — | — | — | |
| Table-R1Backbone=Qwen-2.5-Coder-14B2025.09 | — | — | — | — | 11.8 | 14.08 | — | — | — | |
| TableLLMBackbone=Qwen-2.5-Coder-7B2025.09 | — | — | — | — | 11.99 | 16.75 | — | — | — | |
| TableLLMBackbone=Qwen-2.5-Coder-14B2025.09 | — | — | — | — | 20.99 | 28.88 | — | — | — |