Text-to-SQL on BiomedSQL (test)
90Execution Accuracy (EX)Domain Expert
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Domain Expert2025.05 | 90 | 90 | 95 | — | — | |
| BMSQL-GPT-o3-miniparadigm=BMSQL, backbone=GPT-o3-mini, # Tokens=39,4702025.05 | 62.6 | 69.2 | 83.1 | 38 | 2.6 | |
| DAIL-SQL-GPT-o3-miniparadigm=DAIL-SQL, backbone=GPT-o3-mini, # Tokens=3,3182025.05 | 61.2 | 63.6 | 81.4 | 42.1 | 0 | |
| BMSQL-GPT-4oparadigm=BMSQL, backbone=GPT-4o, # Tokens=32,8192025.05 | 60.4 | 67.2 | 79.8 | 64.5 | 4.9 | |
| Claude-4.5-Opus# Tokens=4,1972025.05 | 58.4 | 59.7 | 80.6 | 35.8 | 0 | |
| Gemini-3-Pro# Tokens=3,1362025.05 | 58.1 | 62.4 | 81.8 | 52.4 | 0 | |
| ReAct-GPT-o3-miniparadigm=ReAct, backbone=GPT-o3-mini, # Tokens=13,3172025.05 | 56.2 | 64.8 | 73.6 | 13.2 | 0 | |
| BMSQL-Gemini*paradigm=BMSQL, backbone=Gemini-2.0-Flash, # Tokens=22,0452025.05 | 55.9 | 61.3 | 84.6 | 32.1 | 0.2 | |
| DAIL-SQL-GPT-4oparadigm=DAIL-SQL, backbone=GPT-4o, # Tokens=3,6242025.05 | 54.8 | 58.1 | 75.5 | 63.4 | 6.6 | |
| GPT-o3-mini# Tokens=3,9422025.05 | 53.5 | 60.4 | 73.3 | 29.4 | 0.2 | |
| DAIL-SQL-Gemini*paradigm=DAIL-SQL, backbone=Gemini-2.0-Flash, # Tokens=3,1852025.05 | 53.1 | 58.8 | 82.8 | 30.6 | 0 | |
| ReAct-GPT-4oparadigm=ReAct, backbone=GPT-4o, # Tokens=14,2862025.05 | 49.6 | 57.9 | 67.2 | 8.9 | 0 | |
| ReAct-Gemini*paradigm=ReAct, backbone=Gemini-2.0-Flash, # Tokens=13,2052025.05 | 48.9 | 56.6 | 60.4 | 10.2 | 0 | |
| GPT-5.2# Tokens=4,5982025.05 | 48.5 | 54.6 | 75.3 | 35.6 | 2.7 | |
| GPT-4o# Tokens=3,6892025.05 | 46.9 | 54.7 | 71.2 | 26.1 | 1.3 | |
| Index-Gemini*paradigm=Index, backbone=Gemini-2.0-Flash, # Tokens=7872025.05 | 46.1 | 48.5 | 54.2 | 59.6 | 8.1 | |
| Claude-3.7-Sonnet# Tokens=3,8052025.05 | 45.4 | 49.8 | 69.8 | 43 | 1.6 | |
| Qwen-2.5-Coder-32B# Tokens=3,6122025.05 | 40.8 | 44.4 | 58.2 | 61 | 15.7 | |
| Llama-3.1-405B# Tokens=3,4562025.05 | 38.1 | 42.5 | 57.9 | 41.7 | 4.6 | |
| Qwen-2.5-Coder-14B# Tokens=3,4532025.05 | 37 | 32.4 | 62.1 | 42.5 | 11 | |
| Llama-3.1-70B# Tokens=3,5472025.05 | 34.4 | 39.8 | 57 | 37 | 6 | |
| Gemini-2.0-Flash# Tokens=3,6922025.05 | 33.7 | 37 | 71.1 | 27.2 | 4.2 | |
| Index-GPT-o3-miniparadigm=Index, backbone=GPT-o3-mini, # Tokens=1,8992025.05 | 27.1 | 30.6 | 44.1 | 47.5 | 2 | |
| Index-GPT-4oparadigm=Index, backbone=GPT-4o, # Tokens=1,1102025.05 | 25.5 | 28.3 | 44.1 | 66.9 | 27.5 |