Text-to-SQL on Spider Lite 2.0
69.3Execution Accuracy (EX)SOMA-SQL
Evaluation Results
| Method | Links | |
|---|---|---|
| SOMA-SQLBackbone LLM=GPT-5.3 Codex2026.06 | 69.3 | |
| SOMA-SQLBackbone LLM=GPT-52026.06 | 65.8 | |
| ProSpyModel=Claude-Opus-4.5, Single Trial=true2026.06 | 60.15 | |
| SOMA-SQLBackbone LLM=Gemma-4-31B2026.06 | 53.7 | |
| SquRL-7BBackbone=Deepseek-R12026.02 | 49.18 | |
| ProSpyModel=Claude + GLM, Single Trial=true2026.06 | 48.63 | |
| AutoLinkBackbone LLM=GPT-5.3 Codex2026.06 | 46.3 | |
| AutoLinkBackbone LLM=GPT-52026.06 | 45.2 | |
| ProSpyModel=Claude + DeepSeek, Single Trial=true2026.06 | 43.14 | |
| ProSpyModel=DeepSeek V3.2, Single Trial=true2026.06 | 41.32 | |
| ProSpyModel=GLM-5, Single Trial=true2026.06 | 38.21 | |
| AutoLinkBackbone LLM=Gemma-4-31B2026.06 | 37.7 | |
| ReForceModel=GPT-o3, Single Trial=false2026.06 | 36.56 | |
| AutoLinkModel=DeepSeek-R1, Single Trial=true2026.06 | 34.92 | |
| DeepEye-SQLBackbone LLM=GPT-5.3 Codex2026.06 | 33.6 | |
| LinkAlignBackbone=Deepseek-R12026.02 | 33.09 | |
| RSL-SQLBackbone=Deepseek-R12026.02 | 33.09 | |
| LinkAlignModel=DeepSeek-R1, Single Trial=false2026.06 | 33.09 | |
| RSL-SQLModel=GPT-o3, Single Trial=false2026.06 | 33.09 | |
| ReForceModel=o4-mini, Single Trial=false2026.06 | 31.99 | |
| DeepEye-SQLBackbone LLM=GPT-52026.06 | 31.4 | |
| RSL-SQLModel=DeepSeek-R1, Single Trial=false2026.06 | 30.53 | |
| Spider-AgentModel=Claude-4-Sonnet, Single Trial=true2026.06 | 27.79 | |
| DeepEye-SQLBackbone LLM=Gemma-4-31B2026.06 | 24 | |
| Agentic SQL + CSMR + ATRRL Data=BIRD + Spider, Base Model=OmniSQL-7B, Config=Cold-Start Model, Decoding Strategy=8-time sampling (majority voting)2026.03 | 17.7 | |
| GPT-4oDecoding Strategy=8-time sampling (majority voting)2026.03 | 15.6 | |
| Deepseek-V3Decoding Strategy=8-time sampling (majority voting)2026.03 | 15.6 | |
| Arctic-Text2SQL-R1-7BRL Data=BIRD + SPIDER + Gretel*, Base Model=OmniSQL-7B, Decoding Strategy=8-time sampling (majority voting)2026.03 | 15.6 | |
| SQL-R1 + OmniSQL-7BRL Data=SynSQL-Complex-5K, Base Model=OmniSQL-7B, Decoding Strategy=8-time sampling (majority voting)2026.03 | 14.8 | |
| Single-Turn GRPORL Data=BIRD + Spider, Base Model=OmniSQL-7B, Config=Cold-Start Model, Decoding Strategy=8-time sampling (majority voting)2026.03 | 14.8 | |
| Spider-AgentModel=DeepSeek-R1, Single Trial=true2026.06 | 13.71 | |
| OmniSQL-7BBase Model=Qwen2.5-Coder-7B, Decoding Strategy=8-time sampling (majority voting)2026.03 | 12.6 | |
| Agentic SQL + CSMR + ATRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=CSMR, Decoding Strategy=8-time sampling (majority voting)2026.03 | 8.8 | |
| ATRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Config=Ablation Study, Decoding Strategy=8-time sampling (majority voting)2026.03 | 8.8 | |
| Agentic SQL + CSMRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=CSMR, Decoding Strategy=8-time sampling (majority voting)2026.03 | 8.1 | |
| ATR w/ Step-wise UpdateRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Config=Ablation Study, Decoding Strategy=8-time sampling (majority voting)2026.03 | 8.1 | |
| Agentic SQL + ATRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=Binary, Decoding Strategy=8-time sampling (majority voting)2026.03 | 7.4 | |
| ATR w/ Symmetric MatrixRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Config=Ablation Study, Decoding Strategy=8-time sampling (majority voting)2026.03 | 7.4 | |
| Single-Turn GRPORL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=Binary, Decoding Strategy=8-time sampling (majority voting)2026.03 | 6.6 | |
| Single-Turn GRPO + CSMRRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=CSMR, Decoding Strategy=8-time sampling (majority voting)2026.03 | 6.6 | |
| Agentic SQLRL Data=BIRD, Base Model=Qwen2.5-7B-Instruct, Reward=Binary, Decoding Strategy=8-time sampling (majority voting)2026.03 | 5.9 | |
| DailSQLBackbone=GPT-4o2026.02 | 5.68 | |
| Qwen2.5-7B-InstructDecoding Strategy=8-time sampling (majority voting)2026.03 | 5.1 | |
| CHESSBackbone=GPT-4o2026.02 | 3.83 | |
| Qwen2.5-Coder-7BDecoding Strategy=8-time sampling (majority voting)2026.03 | 2.2 | |
| DIN-SQLBackbone=GPT-4o2026.02 | 1.46 |