SQL Semantic Validation on Spider
63.21AUPRCPrompt
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PromptBase=GPT-o4-mini2025.12 | 63.21 | 75.81 | |
| HEROSQLBase=Qwen3-4b2025.12 | 53.52 | 69.78 | |
| HEROSQLBase=Qwen3-0.6b2025.12 | 51.92 | 67.32 | |
| HEROSQLBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 51.92 | 67.32 | |
| HEROSQLBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 51.11 | 69.26 | |
| HEROSQL - w/o AST & LPBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 50.39 | 67.82 | |
| HEROSQL - w/o ASTBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 50.03 | 67.29 | |
| HEROSQL - w/o NDABackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 49.73 | 66.19 | |
| HEROSQL - w/o AST & LPBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 49.28 | 61.15 | |
| HEROSQL - w/o NDABackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 48.91 | 59.97 | |
| PromptBase=Claude-haiku-4-52025.12 | 48.72 | 65.8 | |
| PromptBase=GPT-4o2025.12 | 47.68 | 66.37 | |
| HEROSQL - w/o ASTBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 46.81 | 62.73 | |
| TEDBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 43.9 | 62.84 | |
| PromptBase=Qwen3-0.6b2025.12 | 40.86 | 58.94 | |
| PromptBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 40.86 | 58.94 | |
| CoTBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 39.58 | 50.98 | |
| COVEBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 38.09 | 54.78 | |
| ConfScoreBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 35.77 | 50.18 | |
| CoTBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 34.28 | 49.96 | |
| PromptBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 34.26 | 49.75 | |
| COVEBackbone=Gemma-3-0.3B, Domain Setting=In-Domain2025.12 | 34.25 | 49.78 | |
| ConfScoreBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 34.13 | 50.65 | |
| TEDBackbone=Qwen3-0.6B, Domain Setting=In-Domain2025.12 | 33.14 | 46.63 |