Text-to-SQL on BIRD (dev)
77.84Execution Accuracy (EA)MARS-SQL
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MARS-SQLSize=7B each, # agents=3, Thinking?=Yes, Training set=Bird2025.11 | 77.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SIRIUS-SQLModel=SIRIUS-32B + Gemini-3.1 Pro2026.05 | 75.88 | — | — | — | — | — | — | — | — | — | 80.54 | 70.26 | 64.14 | — | — | — | — | — | — | |
| SIRIUS-SQLModel=SIRIUS-32B + Qwen3-235B2026.05 | 75.3 | — | — | — | — | — | — | — | — | — | 80 | 69.4 | 64.14 | — | — | — | — | — | — | |
| CHASELLM Calls (Avg.)=100+, Cost ($)=0.44, Fine-tuning=true2025.05 | 74.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQL + Gemini# agents=4, Thinking?=Yes2025.11 | 74.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Agentar-Scale-SQLModel=Agentar-32B + Gemini2026.05 | 74.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SIRIUS-SQLModel=SIRIUS-32B + Qwen3-Coder-30B2026.05 | 74.51 | — | — | — | — | — | — | — | — | — | 79.24 | 68.97 | 62.07 | — | — | — | — | — | — | |
| CHASE-SQLbackbone=Gemini2024.11 | 74.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQLBackbone=Gemini2025.07 | 74.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSAIRbackbone=GPT-4o2024.11 | 74.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSAIRBackbone=GPT-4o2025.07 | 74.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LongData-SQLModel=N/A2026.05 | 74.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepEye-SQLModel=Qwen3-Coder-30B2026.05 | 73.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XiYan-SQL2024.11 | 73.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XiYan-SQL2025.07 | 73.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XiYan-SQL# agents=3, Thinking?=No2025.11 | 73.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XiYan-SQLModel=XiYanSQL-32B + GPT-4o2026.05 | 73.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XiYan-SQL2026.05 | 73.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOMA-SQLBackbone LLM=GPT-5.3 Codex2026.06 | 73.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQLMethod category=Published, Backbone=Gemini 1.52024.10 | 73.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQLModel=Gemini 1.52026.05 | 73.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQLBase=Gemini-1.5, Size=-, Multiple calls=true2025.05 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQL2026.05 | 73 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQLBase Model=Gemini-1.5-Pro, Method Category=Prompt-based2026.04 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EXPO-SQLBase Model=Qwen2.5-Coder-14B2026.04 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ExSLbackbone=granite-34b-code2024.11 | 72.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ExSLBackbone=granite-34b-code2025.07 | 72.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ErrorLLMBackbone (SQL Generator)=OpenSearch-SQL, Refinement Backbone=GPT-4o2026.03 | 72.29 | — | — | — | — | — | — | — | — | — | 76.65 | 65.52 | 66.21 | — | — | — | — | — | — | |
| Reasoning SQLSize=14B, # agents=1, Thinking?=Yes, Training set=Bird2025.11 | 72.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Reasoning-SQLModel=Qwen2.5-Coder-14B2026.05 | 72.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Insights AIMethod category=Undisclosed2024.10 | 72.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Insights AI2024.11 | 72.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Insights AI2025.07 | 72.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AskDataMethod category=Undisclosed, Backbone=GPT-4o2024.10 | 72.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AskDatabackbone=GPT-4o2024.11 | 72.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SIRIUSModel=SIRIUS-32B2026.05 | 71.97 | — | — | — | — | — | — | — | — | — | 76.22 | 67.67 | 58.62 | — | — | — | — | — | — | |
| ErrorLLMBackbone (SQL Generator)=OpenSearch-SQL, Refinement Backbone=DeepSeek-V32026.03 | 71.9 | — | — | — | — | — | — | — | — | — | 76.43 | 65.09 | 64.83 | — | — | — | — | — | — | |
| OpenSearch-SQLBackbone (SQL Generator)=OpenSearch-SQL2026.03 | 71.38 | — | — | — | — | — | — | — | — | — | 75.46 | 64.66 | 66.9 | — | — | — | — | — | — | |
| EXPO-SQLBase Model=Qwen2.5-Coder-7B2026.04 | 71.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepEye-SQLModel=Qwen3-235B2026.05 | 71.12 | — | — | — | — | — | — | — | — | — | 76.22 | 64.87 | 59.31 | — | — | — | — | — | — | |
| SOMA-SQLBackbone LLM=GPT-52026.06 | 71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Arctic-Text2SQL-R1Size=32B, # agents=1, Thinking?=Yes, Training set=Bird+Spider2025.11 | 70.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DecoSearchLearning Approach=Open-source, Model Avail.=–2026.06 | 70.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Arctic-Text2SQL-R1Model=Arctic-Text2SQL-R1-32B2026.05 | 70.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Reward-SQLCategory=Our Approach, Base Model=Qwen3-8B, Decoding Strategy=PRM@32 (Best-of-N)2025.05 | 70.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAC-SQLBackbone=GPT-4, Oracle Schema=true2023.12 | 70.28 | 62.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Arctic-SQL-R1Base Model=Qwen2.5-Coder-14B, Method Category=RL-based2026.04 | 70.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SOMA-SQLBackbone LLM=Gemma-4-31B2026.06 | 69.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHESSLearning Approach=Training-free, Model Avail.=×2026.06 | 69.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHASE-SQLMethod category=Published, Backbone=Claude 3.5 Sonnet2024.10 | 69.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MCTS-SQLBase Model=GPT-42026.05 | 69.4 | 66.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MCTSBackbone=GPT-4o2026.05 | 69.4 | — | — | — | — | — | — | — | — | — | 74.32 | 65.17 | 51.48 | — | — | — | — | — | — | |
| OpenSearch-v2Method category=Undisclosed, Backbone=GPT-4o2024.10 | 69.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenSearch-SQLversion=v2, backbone=GPT-4o2024.11 | 69.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenSearch-SQLVersion=v2, Backbone=GPT-4o2025.07 | 69.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenSearch-SQL+ GPT-4o# agents=4, Thinking?=No2025.11 | 69.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenSearch-SQLModel=GPT-4o2026.05 | 69.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| N-repLLM Calls (Typical)=8, LLM Calls (Avg.)=10.9, Tokens (K)=32.0, Cost ($)=0.0392025.05 | 69.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FineStepModel Size=30B-A3B2026.05 | 69.23 | 73.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Reward-SQLBase Model=Qwen2.5-Coder-7B, Method Category=RL-based2026.04 | 68.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Arctic-SQL-R1Base Model=Qwen2.5-Coder-7B, Method Category=RL-based2026.04 | 68.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ORM-based Best-of-NN=322026.06 | 68.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 5.01 | — | |
| SQLFixAgentBackbone (SQL Generator)=OpenSearch-SQL, Refinement Method=SQLFixAgent, Refinement Backbone=GPT-4o2026.03 | 68.77 | — | — | — | — | — | — | — | — | — | 74.05 | 60.34 | 62.07 | — | — | — | — | — | — | |
| FineStepModel Size=4B2026.05 | 68.51 | 71.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FineStep-4BParameters=4B2026.05 | 68.51 | 71.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepEye-SQLLearning Approach=Training-free, Model Avail.=×2026.06 | 68.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Reward-SQLCategory=Our Approach, Base Model=Qwen3-8B, Decoding Strategy=PRM@8 (Best-of-N)2025.05 | 68.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRPOModel Size=30B-A3B2026.05 | 68.32 | 70.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHESS2024.11 | 68.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHESS2025.07 | 68.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CHESSLLM Calls (Avg.)=31.83, Tokens (K)=319.9, Cost ($)=0.462025.05 | 68.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ExCoTBase Model=Qwen2.5-Coder-32B, Method Category=RL-based2026.04 | 68.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PURPLE-REDMethod category=Undisclosed, Backbone=GPT-4o2024.10 | 68.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PURPLE + REDbackbone=GPT-4o2024.11 | 68.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PURPLE + REDBackbone=GPT-4o2025.07 | 68.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MTIR-SQLBase Model=Qwen3-14B2026.05 | 68.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepEye-SQLBackbone LLM=GPT-5.3 Codex2026.06 | 68.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Reasoning-SQLCategory=RL-based Methods, Base Model=Qwen2.5-7B, Decoding Strategy=Vote@N2025.05 | 68.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSPOModel Size=4B2026.05 | 68.06 | 69.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FineStepModel Size=8B2026.05 | 68.06 | 70.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FineStep-8BParameters=8B2026.05 | 68.06 | 70.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini-3.1 ProModel=Gemini-3.1 Pro2026.05 | 68.06 | — | — | — | — | — | — | — | — | — | 74.27 | 60.78 | 51.72 | — | — | — | — | — | — | |
| ArcwiseMethod category=Undisclosed, Backbone=GPT-4o2024.10 | 67.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GIGPOModel Size=4B2026.05 | 67.99 | 70.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Alpha-SQLBackbone=XiYanSQL, Size=7B2026.02 | 67.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KE-ICLLLM=Gemini-Pro-1.52026.05 | 67.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DAPOModel Size=4B2026.05 | 67.67 | 69.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SquRL-7Bworkflow=dynamic, number of workflows=5, aggregation=majority voting, parameter scale=7B2026.02 | 67.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CodeSLLM=Gemini-Pro-1.5, Setting=ICL2026.05 | 67.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Progress-SQL-14B(Ours)Reward=Progressive2026.06 | 67.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ExSLMethod category=Undisclosed, Backbone=granite-34b-code2024.10 | 67.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-4-Maverick-17BLearning Approach=Open-source, Model Avail.=✓2026.06 | 67.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistilleryMethod category=Published, Backbone=GPT-4o2024.10 | 67.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Distillerybackbone=GPT-4o2024.11 | 67.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SQL-O1Backbone=XiYanSQL, Size=7B, tuning=true2026.02 | 67.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistilleryBackbone=GPT-4o2025.07 | 67.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Distillery+GPT-4o-finetuneParams.=NA2024.10 | 67.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SQL-R1Base Model=Qwen2.5-Coder-14B, Method Category=RL-based2026.04 | 67.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| No-TrainModel Size=30B-A3B2026.05 | 67.08 | 69.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OmniSQLModel=OmniSQL-32B2026.05 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |