SQL Semantic Validation on BIRD
80.36AUPRCPrompt
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PromptBase=GPT-o4-mini2025.12 | 80.36 | 79.25 | |
| PromptBase=Claude-haiku-4-52025.12 | 75.87 | 76.15 | |
| PromptBase=GPT-4o2025.12 | 73.56 | 73.45 | |
| HEROSQLBase=Qwen3-4b2025.12 | 71.18 | 64.17 | |
| HEROSQLBase=Qwen3-0.6b2025.12 | 67.39 | 61.51 | |
| HEROSQLBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 67.39 | 61.51 | |
| HEROSQLBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 67.1 | 63.48 | |
| HEROSQL - w/o AST & LPBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 66.71 | 61.38 | |
| HEROSQL - w/o ASTBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 64.7 | 60.44 | |
| HEROSQL - w/o NDABackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 64.39 | 60.56 | |
| HEROSQL - w/o ASTBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 64.29 | 55.49 | |
| HEROSQL - w/o NDABackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 62.7 | 59.25 | |
| HEROSQL - w/o AST & LPBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 62.66 | 59.23 | |
| CoTBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 61.01 | 55.31 | |
| PromptBase=Qwen3-0.6b2025.12 | 60.85 | 57.25 | |
| PromptBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 60.85 | 57.25 | |
| TEDBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 60.36 | 57.65 | |
| COVEBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 57.67 | 54.4 | |
| ConfScoreBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 56.59 | 51.43 | |
| TEDBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 56.28 | 52.37 | |
| COVEBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 55.02 | 50.46 | |
| ConfScoreBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 54.79 | 50.53 | |
| CoTBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 54.72 | 49.84 | |
| PromptBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 54.13 | 48.47 |