Text-to-SQL on Science Benchmark
59.53Execution AccuracyKE-ICL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| KE-ICLLLM=GPT-4o2026.05 | 59.53 | — | |
| GPT-4-TurboDecoding=Maj@82025.05 | 59.5 | — | |
| GPT-4-TurboDecoding=Greedy2025.05 | 59.2 | — | |
| Qwen2.5-72B-InstructDecoding=Maj@82025.05 | 58.2 | — | |
| DeepSeek-V3 (671B MoE)Decoding=Maj@82025.05 | 57.9 | — | |
| KE-RLLLM=Qwen2.5-Coder-7B2026.05 | 56.5 | — | |
| GPT-4oDecoding=Maj@82025.05 | 56.2 | — | |
| DeepSeek-V3 (671B MoE)Decoding=Greedy2025.05 | 56.2 | — | |
| GPT-4oDecoding=Greedy2025.05 | 55.5 | — | |
| CodeSLLM=GPT-4o, Setting=ICL2026.05 | 54.84 | — | |
| Reward-SQL + Qwen3-8BDecoding=Maj@82025.05 | 54.6 | — | |
| Arctic-SQL + Qwen2.5-7BDecoding=Maj@82025.05 | 53.8 | — | |
| SQL-R1 + Qwen2.5-7BDecoding=Greedy2025.05 | 53.8 | — | |
| DAIL-SQLLLM=GPT-4o, Setting=ICL2026.05 | 53.62 | — | |
| KE-ICLLLM=Gemini-Pro-1.52026.05 | 53.17 | — | |
| Qwen2.5-72B-InstructDecoding=Greedy2025.05 | 52.8 | — | |
| Arctic-SQL + Qwen2.5-7BDecoding=Greedy2025.05 | 52.5 | — | |
| EVOLSQL-7BParameters=7B2026.01 | 51.8 | 61.9 | |
| OmniSQL + Qwen2.5-7BDecoding=Maj@82025.05 | 51.8 | — | |
| SQL-R1 + Qwen2.5-7BDecoding=Maj@82025.05 | 51.8 | — | |
| CodeSLLM=Gemini-Pro-1.5, Setting=ICL2026.05 | 51.54 | — | |
| Qwen2.5-Coder-7B-InstructDecoding=Maj@82025.05 | 51.2 | — | |
| Reward-SQL + Qwen3-8BDecoding=Greedy2025.05 | 50.8 | — | |
| OmniSQL + Qwen2.5-7BDecoding=Greedy2025.05 | 50.5 | — | |
| OmniSQL-7BParameters=7B2026.01 | 50.2 | 61.4 | |
| DAIL-SQLLLM=Gemini-Pro-1.5, Setting=ICL2026.05 | 50.2 | — | |
| KE-RLLLM=Granite-3.1-8B2026.05 | 49.16 | — | |
| KE-RLLLM=Deepseek-Coder-7B2026.05 | 48.32 | — | |
| SQL-GENLLM=Granite-3.1-8B, Setting=RL2026.05 | 46.88 | — | |
| OmniLLM=Qwen2.5-Coder-7B, Setting=RL2026.05 | 46.48 | — | |
| Base ModelBackbone=Qwen2.5-Coder-7B-Instruct2026.01 | 45.2 | 53.4 | |
| Qwen2.5-Coder-7B-InstructDecoding=Greedy2025.05 | 45.2 | — | |
| SQL-GENLLM=Qwen2.5-Coder-7B, Setting=RL2026.05 | 44.12 | — | |
| SFT (BIRD+Spider)Training Data=BIRD+Spider2026.01 | 43.8 | 55.8 | |
| KE-ICLLLM=Qwen2.5-Coder-7B2026.05 | 43.14 | — | |
| Meta-Llama-3.1-8B-InstructDecoding=Maj@82025.05 | 43.1 | — | |
| OmniLLM=Granite-3.1-8B, Setting=RL2026.05 | 41.13 | — | |
| CodeSLLM=Qwen2.5-Coder-7B, Setting=ICL2026.05 | 39.79 | — | |
| OmniLLM=Deepseek-Coder-7B, Setting=RL2026.05 | 39.29 | — | |
| DAIL-SQLLLM=Qwen2.5-Coder-7B, Setting=ICL2026.05 | 38.22 | — | |
| Meta-Llama-3.1-8B-InstructDecoding=Greedy2025.05 | 36.8 | — | |
| KE-ICLLLM=Granite-3.1-8B2026.05 | 36.12 | — | |
| SQL-GENLLM=Deepseek-Coder-7B, Setting=RL2026.05 | 34.62 | — | |
| KE-ICLLLM=Deepseek-Coder-7B2026.05 | 34.44 | — | |
| CodeSLLM=Granite-3.1-8B, Setting=ICL2026.05 | 33.11 | — | |
| DAIL-SQLLLM=Granite-3.1-8B, Setting=ICL2026.05 | 31.89 | — | |
| CodeSLLM=Deepseek-Coder-7B, Setting=ICL2026.05 | 29.76 | — | |
| DAIL-SQLLLM=Deepseek-Coder-7B, Setting=ICL2026.05 | 28.88 | — |