SQL Execution Performance on BIRD n=1534
64.9EM (1T)Oracle
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| OracleSQL generator=GPT-4o-mini2026.01 | 64.9 | 46.5 | 50.8 | |
| JAR@5SQL generator=GPT-4o-mini2026.01 | 61.6 | 35.9 | 42 | |
| ARMSQL generator=GPT-4o-mini2026.01 | 61.6 | 36.4 | 42.4 | |
| CORE-TSQL generator=GPT-4o-mini2026.01 | 60.8 | 38.4 | 43.7 | |
| DRR@5SQL generator=GPT-4o-mini2026.01 | 60.5 | 34.5 | 40.6 | |
| DR@5SQL generator=GPT-4o-mini2026.01 | 60.2 | 33.4 | 39.7 | |
| ReActSQL generator=GPT-4o-mini2026.01 | 60.2 | 36.3 | 41.9 | |
| OracleSQL generator=Gemma-3-4B-Instruct2026.01 | 44.2 | 17 | 23.4 | |
| CORE-TSQL generator=Gemma-3-4B-Instruct2026.01 | 41.2 | 13.1 | 19.7 | |
| ReActSQL generator=Gemma-3-4B-Instruct2026.01 | 39.8 | 11.3 | 18.1 | |
| ARMSQL generator=Gemma-3-4B-Instruct2026.01 | 37.3 | 12 | 18 | |
| DR@5SQL generator=Gemma-3-4B-Instruct2026.01 | 36.7 | 10.8 | 16.9 | |
| JAR@5SQL generator=Gemma-3-4B-Instruct2026.01 | 36.7 | 10.9 | 17 | |
| DRR@5SQL generator=Gemma-3-4B-Instruct2026.01 | 32.9 | 8.6 | 14.3 | |
| OracleSQL generator=Llama-3.2-3B-Instruct2026.01 | 29.6 | 26.6 | 27.3 | |
| CORE-TSQL generator=Llama-3.2-3B-Instruct2026.01 | 17.7 | 18 | 17.9 | |
| ARMSQL generator=Llama-3.2-3B-Instruct2026.01 | 17.1 | 17.7 | 17.6 | |
| DR@5SQL generator=Llama-3.2-3B-Instruct2026.01 | 16 | 14 | 14.5 | |
| ReActSQL generator=Llama-3.2-3B-Instruct2026.01 | 16 | 16.6 | 16.5 | |
| JAR@5SQL generator=Llama-3.2-3B-Instruct2026.01 | 14.9 | 16.6 | 16.2 | |
| DRR@5SQL generator=Llama-3.2-3B-Instruct2026.01 | 14.1 | 12.5 | 12.8 |