Text-to-SQL on BIRD (Model Execution Accuracy)
31.8Execution Accuracy (Llama-8B)Oracle
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OracleRetrieval category=Oracle2026.04 | 31.8 | 53.5 | 50.6 | 58 | 32.7 | 50.9 | |
| ATRRetrieval category=Ours, Retrieval backbone=UAE-Large-V12026.04 | 29 | 48.7 | 43.8 | 53.9 | 25.6 | 49.7 | |
| ATRRetrieval category=Ours, Retrieval backbone=Contriever2026.04 | 28.6 | 49.9 | 45 | 53.3 | 28.8 | 45.1 | |
| JARk=5, Retrieval category=Reranker-based, Retrieval backbone=Contriever2026.04 | 26.8 | 46.4 | 43.4 | 52 | 27.4 | 43.2 | |
| JARk=10, Retrieval category=Reranker-based, Retrieval backbone=Contriever2026.04 | 26.7 | 47.5 | 43 | 53 | 24.8 | 44.9 | |
| JARk=10, Retrieval category=Reranker-based, Retrieval backbone=UAE-Large-V12026.04 | 26.6 | 46.9 | 41.3 | 52.2 | 24.2 | 46.6 | |
| UAE-Large-V1k=10, Retrieval category=Encoder-based, Retrieval backbone=UAE-Large-V12026.04 | 26 | 47.9 | 40.4 | 52.2 | 22.2 | 46.8 | |
| RankZephyrk=10, Retrieval category=LLM-based2026.04 | 25.4 | 47.5 | 39.3 | 49.5 | 24 | 46.2 | |
| JARk=3, Retrieval category=Reranker-based, Retrieval backbone=Contriever2026.04 | 25.4 | 45.3 | 40.1 | 49.7 | 23.9 | 44.4 | |
| Murrek=10, Retrieval category=LLM-based2026.04 | 25.3 | 46.4 | 39.4 | 48.4 | 21.2 | 47 | |
| JARk=5, Retrieval category=Reranker-based, Retrieval backbone=UAE-Large-V12026.04 | 25.3 | 45.4 | 39.2 | 48.4 | 23.1 | 44.7 | |
| JARk=3, Retrieval category=Reranker-based, Retrieval backbone=UAE-Large-V12026.04 | 24.8 | 44.7 | 38.7 | 44.9 | 22.6 | 44.3 | |
| Contrieverk=10, Retrieval category=Encoder-based, Retrieval backbone=Contriever2026.04 | 24.3 | 47.4 | 39.1 | 53 | 25.6 | 43.2 | |
| UAE-Large-V1k=5, Retrieval category=Encoder-based, Retrieval backbone=UAE-Large-V12026.04 | 23.9 | 45 | 37.8 | 49.5 | 22 | 43.5 | |
| RankZephyrk=5, Retrieval category=LLM-based2026.04 | 23 | 45 | 38.1 | 46.2 | 23.2 | 45.5 | |
| UAE-Large-V1k=3, Retrieval category=Encoder-based, Retrieval backbone=UAE-Large-V12026.04 | 22.6 | 40.7 | 35.1 | 45.4 | 20.1 | 38.8 | |
| Murrek=5, Retrieval category=LLM-based2026.04 | 22.6 | 44.8 | 37.5 | 45.4 | 20.7 | 44.5 | |
| Contrieverk=5, Retrieval category=Encoder-based, Retrieval backbone=Contriever2026.04 | 22.4 | 44.5 | 37 | 49.8 | 24.1 | 39.7 | |
| RankZephyrk=3, Retrieval category=LLM-based2026.04 | 22.4 | 40.7 | 34.5 | 45.8 | 21.7 | 43.2 | |
| Murrek=3, Retrieval category=LLM-based2026.04 | 21.8 | 44.5 | 34.2 | 44.2 | 20.9 | 42.8 | |
| Contrieverk=3, Retrieval category=Encoder-based, Retrieval backbone=Contriever2026.04 | 21.7 | 40.4 | 35.4 | 46 | 21.2 | 38.9 |