NL-to-SQL on WikiTQ
59.7Execution AccuracyTable-Specialist
Evaluation Results
| Method | Links | |
|---|---|---|
| Table-SpecialistBase Model=GPT-4, Training Strategy=Table-Specialist2024.10 | 59.7 | |
| GPT-4Base Model=GPT-4, Training Strategy=Vanilla2024.10 | 55.9 | |
| Table-SpecialistBase Model=GPT-3.5, Training Strategy=Table-Specialist2024.10 | 51.3 | |
| Table-SpecialistBase Model=Llama3.1-8B, Training Strategy=Table-Specialist2024.10 | 44.9 | |
| GPT-3.5Base Model=GPT-3.5, Training Strategy=Vanilla2024.10 | 42.1 | |
| Llama3.1-8BBase Model=Llama3.1-8B, Training Strategy=Vanilla2024.10 | 30 | |
| Table-SpecialistBase Model=GPT-4, Training Strategy=Table-Specialist2024.10 | 0.55 | |
| ReflectionInference Method=Best-of-N (pass@5), Temperature=0.92026.05 | 0.49 | |
| ReflectionInference Method=MCTS2026.05 | 0.449 | |
| GPT-4Base Model=GPT-4, Training Strategy=Vanilla2024.10 | 0.416 | |
| No MemoryInference Method=Best-of-N (pass@5), Temperature=0.92026.05 | 0.408 | |
| Table-SpecialistBase Model=GPT-3.5, Training Strategy=Table-Specialist2024.10 | 0.404 | |
| No MemoryInference Method=MCTS2026.05 | 0.388 | |
| LiTS-FactInference Method=MCTS2026.05 | 0.388 | |
| Raw SiblingInference Method=MCTS2026.05 | 0.367 | |
| LiTS-FactInference Method=Best-of-N (pass@5), Temperature=0.92026.05 | 0.347 | |
| Raw SiblingInference Method=Beam Search2026.05 | 0.306 | |
| ReActInference Method=Single Trajectory, Temperature=02026.05 | 0.286 | |
| No MemoryInference Method=Beam Search2026.05 | 0.265 | |
| Table-SpecialistBase Model=Llama3.1-8B, Training Strategy=Table-Specialist2024.10 | 0.257 | |
| GPT-3.5Base Model=GPT-3.5, Training Strategy=Vanilla2024.10 | 0.209 | |
| Llama3.1-8BBase Model=Llama3.1-8B, Training Strategy=Vanilla2024.10 | 0.138 |