Accuracy Estimation on Text2SQL Transfer Suite (Spider, BIRD, WikiSQL, SParC, CoSQL, SynSQL-2.5M)
3.41MAEMetaEvaluator
Evaluation Results
| Method | Links | |
|---|---|---|
| MetaEvaluatorUnseen model architecture=Meta-Llama-3-70B2026.05 | 3.41 | |
| MetaEvaluatorUnseen model architecture=XiYanSQL-14B2026.05 | 3.55 | |
| MetaEvaluatorUnseen model architecture=Avg.2026.05 | 3.66 | |
| MetaEvaluatorUnseen model architecture=Ministral-3-14B2026.05 | 3.69 | |
| MetaEvaluatorUnseen model architecture=Qwen2.5-32B2026.05 | 3.76 | |
| MetaEvaluatorUnseen model architecture=gemma-2-2b2026.05 | 3.88 | |
| NL2SQL-BUGSUnseen model architecture=Meta-Llama-3-70B2026.05 | 9.31 | |
| NL2SQL-BUGSUnseen model architecture=Ministral-3-14B2026.05 | 9.47 | |
| NL2SQL-BUGSUnseen model architecture=Avg.2026.05 | 9.61 | |
| NL2SQL-BUGSUnseen model architecture=Qwen2.5-32B2026.05 | 9.68 | |
| NL2SQL-BUGSUnseen model architecture=gemma-2-2b2026.05 | 9.76 | |
| NL2SQL-BUGSUnseen model architecture=XiYanSQL-14B2026.05 | 9.84 | |
| AutoEvalUnseen model architecture=Meta-Llama-3-70B2026.05 | 11.62 | |
| AutoEvalUnseen model architecture=Ministral-3-14B2026.05 | 11.89 | |
| AutoEvalUnseen model architecture=Avg.2026.05 | 12.02 | |
| AutoEvalUnseen model architecture=Qwen2.5-32B2026.05 | 12.05 | |
| AutoEvalUnseen model architecture=gemma-2-2b2026.05 | 12.21 | |
| AutoEvalUnseen model architecture=XiYanSQL-14B2026.05 | 12.33 | |
| PseudoAutoEvalUnseen model architecture=Meta-Llama-3-70B2026.05 | 13.84 | |
| PseudoAutoEvalUnseen model architecture=Ministral-3-14B2026.05 | 14.07 | |
| PseudoAutoEvalUnseen model architecture=Avg.2026.05 | 14.24 | |
| PseudoAutoEvalUnseen model architecture=Qwen2.5-32B2026.05 | 14.31 | |
| PseudoAutoEvalUnseen model architecture=gemma-2-2b2026.05 | 14.42 | |
| PseudoAutoEvalUnseen model architecture=XiYanSQL-14B2026.05 | 14.58 | |
| AGDUnseen model architecture=Meta-Llama-3-70B2026.05 | 14.77 | |
| AGDUnseen model architecture=XiYanSQL-14B2026.05 | 14.96 | |
| AGDUnseen model architecture=Ministral-3-14B2026.05 | 15.04 | |
| AGDUnseen model architecture=Avg.2026.05 | 15.04 | |
| AGDUnseen model architecture=Qwen2.5-32B2026.05 | 15.18 | |
| AGDUnseen model architecture=gemma-2-2b2026.05 | 15.25 | |
| DoCUnseen model architecture=Meta-Llama-3-70B2026.05 | 15.42 | |
| DoCUnseen model architecture=Ministral-3-14B2026.05 | 15.66 | |
| DoCUnseen model architecture=Avg.2026.05 | 15.86 | |
| DoCUnseen model architecture=Qwen2.5-32B2026.05 | 15.97 | |
| DoCUnseen model architecture=gemma-2-2b2026.05 | 16.05 | |
| DoCUnseen model architecture=XiYanSQL-14B2026.05 | 16.18 | |
| ATCUnseen model architecture=Meta-Llama-3-70B2026.05 | 17.21 | |
| ATCUnseen model architecture=Ministral-3-14B2026.05 | 17.52 | |
| ATCUnseen model architecture=Avg.2026.05 | 17.72 | |
| ATCUnseen model architecture=Qwen2.5-32B2026.05 | 17.88 | |
| ATCUnseen model architecture=gemma-2-2b2026.05 | 17.95 | |
| ATCUnseen model architecture=XiYanSQL-14B2026.05 | 18.06 |