Text-to-SQL on Spider 1.0 (dev)
86.75Exact Match AccuracyMAC-SQL
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MAC-SQLBase LLM=GPT-4 (32k)2023.12 | 86.75 | — | — | — | |
| DAIL-SQLBase LLM=GPT-42023.12 | 84.4 | — | — | — | |
| DIN-SQLBase LLM=GPT-42023.12 | 82.8 | — | — | — | |
| C3Base LLM=ChatGPT2023.12 | 81.8 | — | — | — | |
| MAC-SQLBase LLM=GPT-3.5-Turbo (16k)2023.12 | 80.56 | — | — | — | |
| CodeT5+Base Parser=SmBoP, Query Representation=PyDict, Edit Representation=Program, User Interaction=Simulated gold SQL (beam size 3)2023.05 | 80 | 81.2 | — | — | |
| CodeT5Base Parser=SmBoP, Query Representation=PyDict, Edit Representation=Program2023.05 | 78 | 78.5 | — | — | |
| CodeT5*Base Parser=SmBoP, Fine-tuning=Edit programs only, Interpreter=Python2023.05 | 77.3 | 77.6 | — | — | |
| GRAPHIX + PICARD + T5-3BPLM category=Text-to-Text, Backbone=T5-3B, Decoding method=PICARD2023.01 | 77.1 | — | — | — | |
| CodeT5+Base Parser=BRIDGEv2, Query Representation=PyDict, Edit Representation=Program, User Interaction=Simulated gold SQL (beam size 3)2023.05 | 76.6 | 78.1 | — | — | |
| S2SQL + ELECTRAPLM category=Encoder-based, Backbone=ELECTRA2023.01 | 76.4 | — | — | — | |
| MAC-SQLBase LLM=SQL-Llama (7B)2023.12 | 76.25 | — | — | — | |
| PICARD + T5-3BPLM category=Text-to-Text, Backbone=T5-3B, Decoding method=PICARD2023.01 | 75.5 | — | — | — | |
| RASAT + PICARD + T5-3BPLM category=Text-to-Text, Backbone=T5-3B, Decoding method=PICARD2023.01 | 75.3 | — | — | — | |
| LGESQL + ELECTRAPLM category=Encoder-based, Backbone=ELECTRA2023.01 | 75.1 | — | — | — | |
| SMBOP + GRAPPAPLM category=Encoder-based, Backbone=GRAPPA2023.01 | 74.7 | — | — | — | |
| No EditBase Parser=SmBoP2023.05 | 74.6 | 75.3 | — | — | |
| NatSQL + GAPPLM category=Encoder-based, Backbone=GAP2023.01 | 73.7 | — | — | — | |
| RAT-SQL + GraPPaEncoder=GraPPa2020.12 | 73.4 | — | — | — | |
| RATSQLPre-trained Model=GraPPa2021.11 | 73.4 | — | — | — | |
| RATSQL + GRAPPAPLM category=Encoder-based, Backbone=GRAPPA2023.01 | 73.4 | — | — | — | |
| SADGAPre-trained Model=GAP2021.11 | 73.1 | — | — | — | |
| CodeT5+Base Parser=CodeT5, Query Representation=PyDict, Edit Representation=Program, User Interaction=Simulated gold SQL (beam size 3)2023.05 | 73 | 72.9 | — | — | |
| RAT-SQL + STRUGEncoder=STRUG2020.12 | 72.7 | — | — | — | |
| RATSQLPre-trained Model=STRUG2021.11 | 72.6 | — | — | — | |
| CodeT5*Base Parser=BRIDGEv2, Fine-tuning=Edit programs only, Interpreter=Python2023.05 | 72.5 | 73.1 | — | — | |
| CodeT5Base Parser=BRIDGEv2, Query Representation=PyDict, Edit Representation=Program2023.05 | 72.5 | 73 | — | — | |
| ShadowGNNPre-trained Model=RoBERTa2021.11 | 72.3 | — | — | — | |
| RAT-SQL + GAP MODELBackbone=GAP MODEL2020.12 | 71.8 | — | — | — | |
| RATSQLPre-trained Model=GAP2021.11 | 71.8 | — | — | — | |
| UnifiedSKG + T5-3BPLM category=Text-to-Text, Backbone=T5-3B2023.01 | 71.8 | — | — | — | |
| SADGAPre-trained Model=BERT-large2021.11 | 71.6 | — | — | — | |
| RyanSQL V2 + BERTBackbone=BERT2020.12 | 70.6 | — | — | — | |
| No EditBase Parser=BRIDGEv22023.05 | 70.1 | 68.2 | — | — | |
| AuxNet + BARTBackbone=BART2020.12 | 70 | — | — | — | |
| BRIDGE v2 + BERTPLM category=Encoder-based, Backbone=BERT2023.01 | 70 | — | — | — | |
| RAT-SQL V3 + BERTBackbone=BERT2020.12 | 69.7 | — | — | — | |
| RATSQLPre-trained Model=BERT-large2021.11 | 69.7 | — | — | — | |
| RATSQL + BERTPLM category=Encoder-based, Backbone=BERT2023.01 | 69.7 | — | — | — | |
| DuoRATPre-trained Model=BERT-large2021.11 | 69.4 | — | — | — | |
| DeepSeek-R1-250120Method Type=Open-Source, Protocol=Prompt Engineering (PE)2025.11 | 69.4 | — | 69.72 | 69.93 | |
| RATSQL-HPFTPre-trained Model=BERT-large2021.11 | 69.3 | — | — | — | |
| CodeT5*Base Parser=CodeT5, Fine-tuning=Edit programs only, Interpreter=Python2023.05 | 69.2 | 68.4 | — | — | |
| SADGAPre-trained Model=BERT-base2021.11 | 69 | — | — | — | |
| CodeT5Base Parser=CodeT5, Query Representation=PyDict, Edit Representation=Program2023.05 | 69 | 68.2 | — | — | |
| RuCo-CMethod Type=RL based, Parameters=7B, Trainset=BIRD2025.11 | 68.07 | — | 68.15 | 67.33 | |
| GPT-4.1Method Type=Proprietary, Protocol=Prompt Engineering (PE)2025.11 | 67.88 | — | 68.8 | 71.46 | |
| RuCo-CMethod Type=RL based, Parameters=14B, Trainset=BIRD2025.11 | 67.82 | — | 67.4 | 63.69 | |
| RAT-SQL + BART EncoderBackbone=BART Encoder2020.12 | 67.6 | — | — | — | |
| RAT-SQL + BERT (replicate)Backbone=BERT-large2020.12 | 66.5 | — | — | — | |
| o3-miniMethod Type=Proprietary, Protocol=Prompt Engineering (PE)2025.11 | 66.24 | — | 67.68 | 72.32 | |
| Qwen2.5-Coder-14B-InstructMethod Type=SFT based2025.11 | 66.24 | — | 65.87 | 62.37 | |
| RAT-SQL V2 + BERTBackbone=BERT2020.12 | 65.8 | — | — | — | |
| RATSQLPre-trained Model=BERT-base2021.11 | 65.8 | — | — | — | |
| SQL-LlamaParameter Count=7B2023.12 | 65.48 | — | — | — | |
| Kimi-K2Method Type=Open-Source, Protocol=Prompt Engineering (PE)2025.11 | 64.96 | — | 66.55 | 71.88 | |
| SADGA2021.11 | 64.7 | — | — | — | |
| IRNet v2Pre-trained Model=BERT-base2021.11 | 63.9 | — | — | — | |
| Deepseek-V3-241226Method Type=Open-Source, Protocol=Prompt Engineering (PE)2025.11 | 62.9 | — | 64.67 | 71.04 | |
| RATSQL2021.11 | 62.7 | — | — | — | |
| No EditBase Parser=CodeT52023.05 | 62.7 | 63.6 | — | — | |
| IRNetPre-training=BERT2019.09 | 61.9 | — | — | — | |
| IRNet + BERTBackbone=BERT2020.12 | 61.9 | — | — | — | |
| GAZP + BERTPLM category=Encoder-based, Backbone=BERT2023.01 | 59.1 | — | — | — | |
| DeepSeek-Coder-6.7B-InstructMethod Type=Open-Source, Protocol=Prompt Engineering (PE)2025.11 | 59 | — | 59.6 | 61.79 | |
| GNNPre-trained Model=Bertrand-DR2021.11 | 57.9 | — | — | — | |
| Editing-Based SQL Query GenerationPre-training=BERT2019.09 | 57.6 | — | — | — | |
| EditSQL + BERTBackbone=BERT2020.12 | 57.6 | — | — | — | |
| EditSQLPre-trained Model=BERT-base2021.11 | 57.6 | — | — | — | |
| Qwen2.5-Coder-14B-InstructMethod Type=Open-Source, Protocol=Prompt Engineering (PE)2025.11 | 57.54 | — | 59.68 | 68.5 | |
| Qwen2.5-Coder-7B-InstructMethod Type=SFT based2025.11 | 57.54 | — | 56.36 | 43.89 | |
| IRNet v22021.11 | 55.4 | — | — | — | |
| Qwen2.5-Coder-7B-InstructMethod Type=Open-Source, Protocol=Prompt Engineering (PE)2025.11 | 53.71 | — | 56.05 | 66.64 | |
| IRNetPre-training=GloVe2019.09 | 53.2 | — | — | — | |
| Global-GNN2020.12 | 52.7 | — | — | — | |
| Global-GNN2021.11 | 52.7 | — | — | — | |
| GNN2019.09 | 40.7 | — | — | — | |
| GNN2021.11 | 40.7 | — | — | — | |
| Editing-Based SQL Query GenerationPre-training=GloVe2019.09 | 36.4 | — | — | — | |
| Leerecursive decoding=true2019.09 | 28.5 | — | — | — | |
| SyntaxSQLNetdata augmentation=true, table-aware=true, history-aware=true2018.10 | 24.8 | — | — | — | |
| SyntaxSQLNetdata augmentation=true2019.09 | 24.8 | — | — | — | |
| SyntaxSQLNetdata augmentation=false, table-aware=true, history-aware=true2018.10 | 18.9 | — | — | — | |
| SyntaxSQLNetdata augmentation=false2019.09 | 18.9 | — | — | — | |
| SyntaxSQLNetdata augmentation=false, table-aware=false, history-aware=true2018.10 | 15.9 | — | — | — | |
| SQLNet2018.10 | 10.9 | — | — | — | |
| SQLNet2019.09 | 10.9 | — | — | — | |
| TypeSQL2018.10 | 8 | — | — | — | |
| SyntaxSQLNetdata augmentation=false, table-aware=false, history-aware=false2018.10 | 6.1 | — | — | — | |
| Seq2Seq+Copying2018.10 | 4.1 | — | — | — | |
| Seq2Seq2018.10 | 1.9 | — | — | — | |
| Seq2Seq+Attention2018.10 | 1.8 | — | — | — | |
| C3 + ChatGPTMode=Zero-Shot2023.07 | — | 81.8 | — | — | |
| ChatGPT-SQL + ChatGPTMode=Zero-Shot2023.07 | — | 72.3 | — | — | |
| DIN-SQL + GPT-4Mode=Few-Shot2023.07 | — | 82.8 | — | — | |
| Graphix-3B + PICARDMode=Fine-tuning2023.07 | — | 81 | — | — | |
| RATSQL + GAP + NatSQLMode=Fine-tuning2023.07 | — | 75 | — | — | |
| RESDSQL-3B + NatSQLMode=Fine-tuning2023.07 | — | 84.1 | — | — | |
| SC-Prompt + T5-3BMode=Few-Shot2023.07 | — | 81.1 | — | — | |
| T5-3B + PICARDMode=Fine-tuning2023.07 | — | 79.3 | — | — |