Text-to-SQL on BIRD (Accuracy)
69.1AccuracyAgentic SQL + CSMR + ATR
Evaluation Results
| Method | Links | |
|---|---|---|
| Agentic SQL + CSMR + ATRRL Data=BIRD + Spider, Base Model=OmniSQL-7B, Config=Cold-Start Model, Decoding Strategy=Greedy2026.03 | 69.1 | |
| Arctic-Text2SQL-R1-7BRL Data=BIRD + SPIDER + Gretel*, Base Model=OmniSQL-7B, Decoding Strategy=Greedy2026.03 | 67.6 | |
| Single-Turn GRPORL Data=BIRD + Spider, Base Model=OmniSQL-7B, Config=Cold-Start Model, Decoding Strategy=Greedy2026.03 | 67.4 | |
| SQL-R1 + OmniSQL-7BRL Data=SynSQL-Complex-5K, Base Model=OmniSQL-7B, Decoding Strategy=Greedy2026.03 | 66.6 | |
| GPT-4oDecoding Strategy=Greedy2026.03 | 64.4 | |
| Agentic SQL + CSMR + ATRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=CSMR, Decoding Strategy=Greedy2026.03 | 64.2 | |
| ATRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Config=Ablation Study, Decoding Strategy=Greedy2026.03 | 64.2 | |
| OmniSQL-7BBase Model=Qwen2.5-Coder-7B, Decoding Strategy=Greedy2026.03 | 64.1 | |
| Reasoning-SQL-7BRL Data=BIRD, Base Model=Qwen2.5-Coder-7B, Decoding Strategy=Greedy2026.03 | 64 | |
| Agentic SQL + ATRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=Binary, Decoding Strategy=Greedy2026.03 | 63.6 | |
| SQL-R1 + Qwen2.5-Coder-7BRL Data=SynSQL-Complex-5K, Base Model=Qwen2.5-Coder-7B, Decoding Strategy=Greedy2026.03 | 63.1 | |
| MTIR-SQL-4BRL Data=BIRD + SPIDER, Base Model=Qwen3-4B, Decoding Strategy=Greedy2026.03 | 63.1 | |
| Deepseek-V3Decoding Strategy=Greedy2026.03 | 62.5 | |
| Agentic SQL + CSMRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=CSMR, Decoding Strategy=Greedy2026.03 | 62.5 | |
| ATR w/ Step-wise UpdateRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Config=Ablation Study, Decoding Strategy=Greedy2026.03 | 61.3 | |
| Agentic SQLRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=Binary, Decoding Strategy=Greedy2026.03 | 61.1 | |
| ATR w/ Symmetric MatrixRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Config=Ablation Study, Decoding Strategy=Greedy2026.03 | 60.1 | |
| Single-Turn GRPO + CSMRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=CSMR, Decoding Strategy=Greedy2026.03 | 59.4 | |
| Single-Turn GRPORL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=Binary, Decoding Strategy=Greedy2026.03 | 58.5 | |
| Qwen2.5-Coder-7BDecoding Strategy=Greedy2026.03 | 58.2 | |
| Qwen2.5-7B-InstructDecoding Strategy=Greedy2026.03 | 47.5 | |
| Table-SpecialistBase Model=GPT-4, Training Strategy=Table-Specialist2024.10 | 47.5 | |
| GPT-4Base Model=GPT-4, Training Strategy=Vanilla2024.10 | 43 | |
| Table-SpecialistBase Model=GPT-3.5, Training Strategy=Table-Specialist2024.10 | 40.4 | |
| GPT-3.5Base Model=GPT-3.5, Training Strategy=Vanilla2024.10 | 31.7 | |
| Table-SpecialistBase Model=Llama3.1-8B, Training Strategy=Table-Specialist2024.10 | 26.1 | |
| Llama3.1-8BBase Model=Llama3.1-8B, Training Strategy=Vanilla2024.10 | 22.5 |