Text-to-OverpassQL on OverpassNL (test)
40.6EMOSMT-base
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OSMT-baseSetting=+TRA +comments2025.12 | 40.6 | — | — | — | — | 54.6 | |
| Claude-4-sonnetSetting=retrieval-sBERT+TRA, Evaluation Protocol=5-Shot In-Context Learning2025.12 | 39.8 | — | — | — | — | 53.7 | |
| GPT4Setting=retrieval-sBERT2025.12 | 38.9 | — | — | — | — | 53 | |
| DeepSeek-V3Setting=retrieval-sBERT+TRA, Evaluation Protocol=5-Shot In-Context Learning2025.12 | 37.6 | — | — | — | — | 51.2 | |
| Claude-4-sonnetSetting=retrieval-sBERT, Evaluation Protocol=5-Shot In-Context Learning2025.12 | 37.4 | — | — | — | — | 52.2 | |
| DeepSeek-V3Setting=retrieval-sBERT, Evaluation Protocol=5-Shot In-Context Learning2025.12 | 36.4 | — | — | — | — | 50.1 | |
| OSMT-baseSetting=+comments2025.12 | 35 | — | — | — | — | 47.4 | |
| OverpassT5Setting=+comments2025.12 | 33.9 | — | — | — | — | 46.3 | |
| OSMT-baseSetting=+TRA +comments, #Params=582M2025.12 | 24.5 | 76.3 | 73.6 | 73.4 | 74.4 | — | |
| OSMT-baseSetting=+TRA, #Params=582M2025.12 | 24.3 | 75.8 | 71.7 | 71.2 | 72.9 | — | |
| Claude-4-sonnetSetting=5-Shot retrieval-sBERT+TRA, #Params=N/A2025.12 | 22.1 | 75.1 | 72.1 | 72.5 | 73.2 | — | |
| DeepSeek-V3Setting=5-Shot retrieval-sBERT+TRA, #Params=671B2025.12 | 21.8 | 74.5 | 71.1 | 72.3 | 72.6 | — | |
| GPT-4oSetting=5-Shot retrieval-sBERT+TRA, #Params=~200B2025.12 | 21.7 | 74.5 | 71.2 | 71.5 | 72.4 | — | |
| GPT-4oSetting=5-Shot retrieval-sBERT, #Params=~200B2025.12 | 21.5 | 74.1 | 68.6 | 71.6 | 71.4 | — | |
| DeepSeek-V3Setting=5-Shot retrieval-sBERT, #Params=671B2025.12 | 21.4 | 73.8 | 67.6 | 71.7 | 71 | — | |
| OSMT-baseSetting=+comments, #Params=582M2025.12 | 21.3 | 75.2 | 65.5 | 73.2 | 71.3 | — | |
| GPT-4.1Setting=5-Shot retrieval-sBERT+TRA, #Params=N/A2025.12 | 20.7 | 74.1 | 71.7 | 71.7 | 72.5 | — | |
| Claude-4-sonnetSetting=5-Shot retrieval-sBERT, #Params=N/A2025.12 | 20.7 | 74.5 | 68.9 | 71.6 | 71.6 | — | |
| OverpassT5Setting=+comments, #Params=582M2025.12 | 20.7 | 74.9 | 66.1 | 72.7 | 71.2 | — | |
| GPT4Setting=retrieval-sBERT, #Params=~1.76T2025.12 | 20.7 | 73.6 | 68.6 | 72 | 71.4 | — | |
| GPT-4.1Setting=5-Shot retrieval-sBERT, #Params=N/A2025.12 | 20.6 | 73.5 | 68.6 | 71.5 | 71.2 | — | |
| OSMT-baseSetting=fine-tuned, #Params=582M2025.12 | 20.6 | 75.7 | 67.6 | 72.8 | 71.4 | — | |
| Qwen3-235BSetting=5-Shot retrieval-sBERT, #Params=235B2025.12 | 20.3 | 72.4 | 66.6 | 70.9 | 70 | — | |
| Qwen2.5-72BSetting=5-Shot retrieval-sBERT, #Params=72B2025.12 | 17.4 | 71.1 | 66.3 | 69.2 | 68.9 | — | |
| LLaMA-3.1-8BSetting=5-Shot retrieval-sBERT, #Params=8B2025.12 | 12.4 | 65.4 | 57.5 | 60.8 | 61.2 | — |