List your company
ResearchFormerly WizWand

Semantic Parsing on GeoQuery (i.i.d.)

93.6Exact Match AccuracyT5 11B (Prompt Tuning)
52.10462.87773.6584.423Dec 13, 2022Apr 14, 2023Aug 15, 2023Dec 16, 2023Apr 16, 2024Aug 17, 2024Dec 18, 2024
Updated 8mo ago

Evaluation results

MethodLinks
T5 11B (Prompt Tuning)Evaluation Protocol=Prompt Tuning, Source=Qiu et al. 2022b2022.1293.6
T5 Base (FT)Evaluation Protocol=Fine-tuned, Source=Qiu et al. 2022a2022.1293.3
T5 Base + CSL-Aug (FT)Evaluation Protocol=Fine-tuned, Data Augmentation=CSL-Aug, Source=Qiu et al. 2022a2022.1293.3
T5 Base (FT)Evaluation Protocol=Fine-tuned, Source=Qiu et al. 2022b2022.1292.9
PaLM 62B (FT)Evaluation Protocol=Fine-tuned, Source=Qiu et al. 2022b2022.1292.5
T5 Large (fine tuned w/o prompts)Evaluation Protocol=Fine-tuned2022.1292.5
SUBSBase Model=BART2023.0691.8
Cover-LS (NoFT)Evaluation Protocol=No Fine-Tuning, Backbone=Codex2022.1291.4
SpanSubBase Model=BART2023.0690.6
T5 (fine tuned w/o prompts)Model=T5-large, Evaluation Protocol=Finetuned, Retrieval Strategy=N/A, Oracle Status=false2022.1290.3
BARTBase Model=BART2023.0690.2
LexSymBase Model=BART2023.0690.2
Cover-UttModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=BM25 (Diversity), Oracle Status=false2022.1289
Top-K (NoFT)Evaluation Protocol=No Fine-Tuning, Backbone=Codex2022.1288.9
Cover-LSModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Local Structure Coverage, Oracle Status=false2022.1288.7
Transformer2024.1288
GECABase Model=BART2023.0687.9
DPPModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Determinantal Point Process, Oracle Status=false2022.1287
RU-Transformer2024.1287
PaLM 540B (ICL)Evaluation Protocol=In-Context Learning, Source=Qiu et al. 2022b2022.1286.8
Top-KModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=BM25, Oracle Status=false2022.1286.3
Top-K (Oracle)Model=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=BM25 over gold programs, Oracle Status=true2022.1286.3
Cover-LS (Oracle)Model=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Oracle LS Coverage, Oracle Status=true2022.1286.3
SpanSubBase Model=LSTM2023.0682.4
LexSymBase Model=LSTM2023.0681.6
SUBSBase Model=LSTM2023.0680.5
SpanParse2023.0678.9
GECABase Model=LSTM2023.0676.8
NQG2024.1276
LSTMBase Model=LSTM2023.0675.2
sDTM2024.1273
RandomModel=Codex, Evaluation Protocol=NoFT, Retrieval Strategy=Random Sampling, Oracle Status=false2022.1253.7