Text-to-OverpassQL on OverpassNL (val)
26.3EMOSMT-base
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| OSMT-baseEvaluation Protocol=Finetuning, #Params.=582M, Setting=+TRA +comments2025.12 | 26.3 | 77 | 75.3 | 74.3 | 75.5 | |
| OSMT-baseEvaluation Protocol=Finetuning, #Params.=582M, Setting=+TRA2025.12 | 25.8 | 76.5 | 74 | 72.8 | 74.4 | |
| OSMT-smallEvaluation Protocol=Finetuning, #Params.=300M, Setting=+TRA +comments2025.12 | 25.6 | 76.4 | 75.3 | 74.2 | 75.3 | |
| OSMT-smallEvaluation Protocol=Finetuning, #Params.=300M, Setting=+TRA2025.12 | 24.6 | 75.4 | 73.2 | 71.9 | 73.5 | |
| GPT-4*Evaluation Protocol=5-Shot In-Context Learning, #Params.=~1.76T, Setting=retrieval-sBERT2025.12 | 23.4 | 75.7 | 69.9 | 74 | 73.2 | |
| GPT-4oEvaluation Protocol=5-Shot In-Context Learning, #Params.=~200B, Setting=retrieval-sBERT2025.12 | 22.8 | 74.7 | 69.1 | 72.4 | 72.1 | |
| OSMT-baseEvaluation Protocol=Finetuning, #Params.=582M, Setting=standard2025.12 | 22.4 | 75.6 | 65.2 | 73.7 | 71.5 | |
| ByT5-base*Evaluation Protocol=Finetuning, #Params.=582M, Setting=+comments2025.12 | 22 | 75.5 | 66 | 73.7 | 71.7 | |
| GPT-4.1Evaluation Protocol=5-Shot In-Context Learning, #Params.=N/A, Setting=retrieval-sBERT2025.12 | 22 | 74.5 | 68.3 | 72.8 | 71.8 | |
| ByT5-base*Evaluation Protocol=Finetuning, #Params.=582M, Setting=standard2025.12 | 21.9 | 75.5 | 65 | 73.8 | 71.4 | |
| DeepSeek-V3Evaluation Protocol=5-Shot In-Context Learning, #Params.=671B, Setting=retrieval-sBERT2025.12 | 21.9 | 74.6 | 68.3 | 72.6 | 71.8 | |
| Qwen3-235BEvaluation Protocol=5-Shot In-Context Learning, #Params.=235B, Setting=retrieval-sBERT2025.12 | 21.7 | 73.3 | 66.3 | 72.1 | 70.6 | |
| Claude-4-sonnetEvaluation Protocol=5-Shot In-Context Learning, #Params.=N/A, Setting=retrieval-sBERT2025.12 | 21.7 | 75.5 | 68.3 | 73 | 72.3 | |
| ByT5-small*Evaluation Protocol=Finetuning, #Params.=300M, Setting=+comments2025.12 | 21 | 75 | 64.6 | 73.4 | 71 | |
| OSMT-smallEvaluation Protocol=Finetuning, #Params.=300M, Setting=standard2025.12 | 20.7 | 74.7 | 63.8 | 73 | 70.5 | |
| ByT5-small*Evaluation Protocol=Finetuning, #Params.=300M, Setting=standard2025.12 | 20.4 | 74.8 | 64.4 | 73.1 | 70.8 | |
| CodeT5-base*Evaluation Protocol=Finetuning, #Params.=220M, Setting=+comments2025.12 | 20.3 | 74.9 | 63.6 | 73.5 | 70.7 | |
| Qwen2.5-72BEvaluation Protocol=5-Shot In-Context Learning, #Params.=72B, Setting=retrieval-sBERT2025.12 | 19.9 | 72.6 | 68 | 70.3 | 70.3 | |
| CodeT5-base*Evaluation Protocol=Finetuning, #Params.=220M, Setting=standard2025.12 | 19.8 | 74.6 | 63.2 | 73 | 70.3 | |
| CodeT5+Evaluation Protocol=Finetuning, #Params.=220M, Setting=+comments2025.12 | 19.2 | 73.4 | 62.9 | 72.1 | 69.5 | |
| CodeT5+Evaluation Protocol=Finetuning, #Params.=220M, Setting=standard2025.12 | 18.6 | 74 | 62.5 | 71.9 | 69.4 | |
| LLaMA-3.1-8BEvaluation Protocol=5-Shot In-Context Learning, #Params.=8B, Setting=retrieval-sBERT2025.12 | 13.7 | 67 | 59.4 | 62.6 | 63 |