Semantic Parsing on GeoQuery (i.i.d.)
93.6Exact Match AccuracyT5 11B (Prompt Tuning)
Evaluation Results
| Method | Links | |
|---|---|---|
| T5 11B (Prompt Tuning)Evaluation Protocol=Prompt Tuning, Source=Qiu et al. 2022b2022.12 | 93.6 | |
| T5 Base (FT)Evaluation Protocol=Fine-tuned, Source=Qiu et al. 2022a2022.12 | 93.3 | |
| T5 Base + CSL-Aug (FT)Evaluation Protocol=Fine-tuned, Data Augmentation=CSL-Aug, Source=Qiu et al. 2022a2022.12 | 93.3 | |
| T5 Base (FT)Evaluation Protocol=Fine-tuned, Source=Qiu et al. 2022b2022.12 | 92.9 | |
| PaLM 62B (FT)Evaluation Protocol=Fine-tuned, Source=Qiu et al. 2022b2022.12 | 92.5 | |
| T5 Large (fine tuned w/o prompts)Evaluation Protocol=Fine-tuned2022.12 | 92.5 | |
| SUBSBase Model=BART2023.06 | 91.8 | |
| Cover-LS (NoFT)Evaluation Protocol=No Fine-Tuning, Backbone=Codex2022.12 | 91.4 | |
| SpanSubBase Model=BART2023.06 | 90.6 | |
| T5 (fine tuned w/o prompts)Model=T5-large, Evaluation Protocol=Finetuned, Retrieval Strategy=N/A, Oracle Status=false2022.12 | 90.3 | |
| BARTBase Model=BART2023.06 | 90.2 | |
| LexSymBase Model=BART2023.06 | 90.2 | |
| Cover-UttModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=BM25 (Diversity), Oracle Status=false2022.12 | 89 | |
| Top-K (NoFT)Evaluation Protocol=No Fine-Tuning, Backbone=Codex2022.12 | 88.9 | |
| Cover-LSModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Local Structure Coverage, Oracle Status=false2022.12 | 88.7 | |
| Transformer2024.12 | 88 | |
| GECABase Model=BART2023.06 | 87.9 | |
| DPPModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Determinantal Point Process, Oracle Status=false2022.12 | 87 | |
| RU-Transformer2024.12 | 87 | |
| PaLM 540B (ICL)Evaluation Protocol=In-Context Learning, Source=Qiu et al. 2022b2022.12 | 86.8 | |
| Top-KModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=BM25, Oracle Status=false2022.12 | 86.3 | |
| Top-K (Oracle)Model=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=BM25 over gold programs, Oracle Status=true2022.12 | 86.3 | |
| Cover-LS (Oracle)Model=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Oracle LS Coverage, Oracle Status=true2022.12 | 86.3 | |
| SpanSubBase Model=LSTM2023.06 | 82.4 | |
| LexSymBase Model=LSTM2023.06 | 81.6 | |
| SUBSBase Model=LSTM2023.06 | 80.5 | |
| SpanParse2023.06 | 78.9 | |
| GECABase Model=LSTM2023.06 | 76.8 | |
| NQG2024.12 | 76 | |
| LSTMBase Model=LSTM2023.06 | 75.2 | |
| sDTM2024.12 | 73 | |
| RandomModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Random Sampling, Oracle Status=false2022.12 | 53.7 |