Text-to-SQL on PolySQL (test)
63.1Execution Accuracy (SQLite)Claude 3.5 Sonnet
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Claude 3.5 Sonnet2026.05 | 63.1 | 50 | 48.8 | 45.2 | 47.2 | 48.4 | 50.5 | |
| GPT-OSSParameters=120B2026.05 | 54.8 | 42.1 | 43.3 | 41.3 | 40.5 | 40.9 | 43.8 | |
| Llama 3.1Parameters=405B2026.05 | 54.8 | 40.9 | 40.5 | 34.5 | 39.3 | 45.2 | 42.5 | |
| Claude 3.5 Haiku2026.05 | 54.8 | 39.3 | 42.1 | 38.1 | 38.5 | 42.1 | 42.5 | |
| DeepSeek V32026.05 | 53.2 | 42.9 | 40.5 | 39.7 | 37.3 | 44.8 | 43.1 | |
| GPT-OSSParameters=20B2026.05 | 53.2 | 41.7 | 41.7 | 34.5 | 38.5 | 39.7 | 41.5 | |
| DeepSeek V2.52026.05 | 52.8 | 40.5 | 38.5 | 43.7 | 40.9 | 42.1 | 43.1 | |
| Qwen 2.5Parameters=72B2026.05 | 52 | 39.3 | 38.5 | 35.3 | 36.1 | 43.7 | 40.8 | |
| Llama 4 Maverick2026.05 | 51.2 | 42.5 | 38.5 | 38.5 | 36.9 | 39.7 | 41.2 | |
| Llama 3.3Parameters=70B2026.05 | 50 | 39.3 | 40.5 | 40.5 | 34.5 | 38.1 | 40.5 | |
| Llama 4 ScoutParameters=17B2026.05 | 47.2 | 34.1 | 36.9 | 32.5 | 32.5 | 40.5 | 37.3 | |
| Mistral Small2026.05 | 46 | 37.3 | 36.1 | 32.5 | 32.1 | 40.5 | 37.4 | |
| DeepSeek CoderParameters=33B2026.05 | 43.7 | 34.9 | 33.7 | 32.1 | 29.8 | 34.5 | 34.8 | |
| Qwen 3Parameters=8B2026.05 | 40.9 | 29.4 | 32.9 | 23.8 | 23.8 | 28.2 | 29.8 | |
| Llama 3.1Parameters=8B2026.05 | 34.5 | 26.2 | 28.2 | 26.6 | 25 | 16.7 | 26.2 | |
| Granite 3.3Parameters=8B2026.05 | 27.8 | 20.6 | 23.8 | 23.8 | 24.2 | 6.3 | 21.1 |