Semantic Parsing on GeoQuery
77.9Template AccuracyGRPO-Composite
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GRPO-CompositeBase Model=Qwen-2.5-7B-Instruct2026.05 | 77.9 | — | 33.76 | |
| GRPO-BinaryBase Model=Qwen-2.5-7B-Instruct2026.05 | 75.27 | — | 32.14 | |
| SFTBase Model=Qwen-2.5-7B-Instruct2026.05 | 73.65 | — | 28.21 | |
| GRPO-BinaryBase Model=Llama-3.1-8B-Instruct2026.05 | 52.48 | — | 45.92 | |
| GRPO-CompositeBase Model=Llama-3.1-8B-Instruct2026.05 | 52.07 | — | 46.17 | |
| SFTBase Model=Llama-3.1-8B-Instruct2026.05 | 50.36 | — | 43.21 | |
| Grammar Prompting# ICL examples=32, # retrieval set=5602023.05 | — | 97.9 | — | |
| Grammar Prompting (w. oracle grammar)# ICL examples=32, # retrieval set=5602023.05 | — | 98.6 | — | |
| Previous Work# ICL examples=32, # retrieval set=5602023.05 | — | 86.1 | — | |
| Standard Prompting# ICL examples=32, # retrieval set=5602023.05 | — | 96.8 | — |