Text-to-SQL on Spider 200 question subset with paraphrase perturbations (dev)
0.894AST Similarity (para)Gemini 3 Pro
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Gemini 3 ProPerturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.894 | 1.53 | 7.9 | 19.5 | |
| Gemini 2.5 FlashPerturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.787 | 2.87 | 15.8 | 39.5 | |
| Claude Sonnet 4.5Perturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.755 | 2.19 | 17.9 | 36.5 | |
| Claude Opus 4.5Perturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.755 | 2.19 | 17.9 | 36.5 | |
| DeepSeek V3.1Perturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.655 | 3.33 | 28 | 51.5 | |
| GPT 4.1 miniPerturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.493 | 6.64 | 46.2 | 77 | |
| GPT 5 miniPerturbation Type=Paraphrase, Protocol=repeated-sampling protocol2026.04 | 0.328 | 13.22 | 62.2 | 90 |