SQL Semantic Validation on EHRSQL
89.28AUPRCHEROSQL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| HEROSQLBase=Qwen3-4b2025.12 | 89.28 | 72.03 | |
| HEROSQLBase=Qwen3-0.6b2025.12 | 89.07 | 69.53 | |
| HEROSQLBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 89.07 | 69.53 | |
| HEROSQL - w/o NDABackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 88.62 | 68.86 | |
| HEROSQL - w/o AST & LPBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 88.23 | 63.24 | |
| HEROSQL - w/o ASTBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 87.34 | 69.2 | |
| PromptBase=GPT-4o2025.12 | 86.78 | 67.04 | |
| PromptBase=GPT-o4-mini2025.12 | 86.38 | 68 | |
| COVEBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 85.53 | 56.54 | |
| HEROSQLBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 85.48 | 65.92 | |
| HEROSQL - w/o ASTBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 85.22 | 65.02 | |
| HEROSQL - w/o AST & LPBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 85.1 | 62.08 | |
| PromptBase=Qwen3-0.6b2025.12 | 84.72 | 58.86 | |
| PromptBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 84.72 | 58.86 | |
| HEROSQL - w/o NDABackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 84.14 | 60.77 | |
| CoTBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 84.13 | 59.61 | |
| PromptBase=Claude-haiku-4-52025.12 | 83.92 | 61.26 | |
| TEDBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 82.35 | 50.76 | |
| CoTBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 81.73 | 54.11 | |
| COVEBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 81.58 | 51.79 | |
| ConfScoreBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 80.66 | 50.39 | |
| PromptBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 78.86 | 44.79 | |
| ConfScoreBackbone=Gemma-3-0.3B, Domain Setting=Out-of-Domain2025.12 | 78.48 | 46.14 | |
| TEDBackbone=Qwen3-0.6B, Domain Setting=Out-of-Domain2025.12 | 76.07 | 41.76 |