Text-to-SQL on BiomedSQL
62.6Execution Accuracy (EX)BMSQL-GPT-o3-mini
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| BMSQL-GPT-o3-miniInteraction Paradigm=BMSQL, LLM Backbone=GPT-o3-mini2025.05 | 62.6 | 69.2 | 83.1 | 38 | 2.6 | 39,470 | |
| DAIL-SQL-GPT-o3-miniInteraction Paradigm=DAIL-SQL, LLM Backbone=GPT-o3-mini2025.05 | 61.2 | 63.6 | 81.4 | 42.1 | 0 | 3,318 | |
| BMSQL-GPT-4oInteraction Paradigm=BMSQL, LLM Backbone=GPT-4o2025.05 | 60.4 | 67.2 | 79.8 | 64.5 | 4.9 | 32,819 | |
| ReAct-GPT-o3-miniInteraction Paradigm=ReAct, LLM Backbone=GPT-o3-mini2025.05 | 56.2 | 64.8 | 73.6 | 13.2 | 0 | 13,317 | |
| BMSQL-Gemini*Interaction Paradigm=BMSQL, LLM Backbone=Gemini-2.0-Flash2025.05 | 55.9 | 61.3 | 84.6 | 32.1 | 0.2 | 22,045 | |
| DAIL-SQL-GPT-4oInteraction Paradigm=DAIL-SQL, LLM Backbone=GPT-4o2025.05 | 54.8 | 58.1 | 75.5 | 63.4 | 6.6 | 3,624 | |
| DAIL-SQL-Gemini*Interaction Paradigm=DAIL-SQL, LLM Backbone=Gemini-2.0-Flash2025.05 | 53.1 | 58.8 | 82.8 | 30.6 | 0 | 3,185 | |
| ReAct-GPT-4oInteraction Paradigm=ReAct, LLM Backbone=GPT-4o2025.05 | 49.6 | 57.9 | 67.2 | 8.9 | 0 | 14,286 | |
| ReAct-Gemini*Interaction Paradigm=ReAct, LLM Backbone=Gemini-2.0-Flash2025.05 | 48.9 | 56.6 | 60.4 | 10.2 | 0 | 13,205 | |
| Index-Gemini*Interaction Paradigm=Index, LLM Backbone=Gemini-2.0-Flash2025.05 | 46.1 | 48.5 | 54.2 | 59.6 | 8.1 | 787 | |
| Index-GPT-o3-miniInteraction Paradigm=Index, LLM Backbone=GPT-o3-mini2025.05 | 27.1 | 30.6 | 44.1 | 47.5 | 2 | 1,899 | |
| Index-GPT-4oInteraction Paradigm=Index, LLM Backbone=GPT-4o2025.05 | 25.5 | 28.3 | 44.1 | 66.9 | 27.5 | 1,110 |