Semantic Parsing on COGS (generalization)
99Accuracy (Generalization)Lear
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Learstructural trees=implicit or explicit use, conjunct order=disregarded or post-processed2023.05 | 99 | — | — | 100 | 99 | 93 | 99 | |
| Q'22Pretrained Transformer=yes, Uses Trees=yes2023.05 | 99 | — | — | — | — | — | — | |
| D'22Pretrained Transformer=yes, Uses Trees=yes2023.05 | 99 | — | — | — | — | — | — | |
| Multiset Tagging and Latent Permutationslexicon mechanism=complex lexicon induction, conjunct order=disregarded or post-processed2023.05 | 93 | — | — | 82 | 91 | 33 | 97 | |
| OursLexicon Induction=true2023.05 | 92 | — | — | 79 | 85 | 9 | 97 | |
| DanglePretrained Transformer=yes (T5)2023.05 | 88 | — | — | 25 | 35 | 0 | 99 | |
| ROBERTA + DANGLE-ENCbackbone=RoBERTa, DANGLE-ENC=true2021.10 | 87.6 | — | 0 | 24.6 | 34.7 | — | — | |
| Edge TransformerPrediction paradigm=graph prediction2021.12 | 87.4 | — | — | — | — | — | — | |
| Danglestructural trees=no, conjunct order=disregarded or post-processed2023.05 | 87 | — | — | 14 | 14 | 8 | 99 | |
| T5-basePretrained Transformer=yes2023.05 | 86 | — | — | 13 | 18 | 0 | 99 | |
| TRANSFORMER + DANGLE-ENCposition_encoding=absolute, DANGLE-ENC=true2021.10 | 85.9 | — | 0 | 11.4 | 5.7 | — | — | |
| T5-BASE2021.10 | 85.9 | — | 0 | 12.5 | 18 | — | — | |
| TRANSFORMERposition_encoding=absolute2021.10 | 85.5 | — | 0 | 3.4 | 8.9 | — | — | |
| TRANSFORMER + DANGLE-ENCposition_encoding=relative, DANGLE-ENC=true2021.10 | 85.4 | — | 0 | 13.8 | 13.5 | — | — | |
| DanglePretrained Transformer=no2023.05 | 85 | — | — | 14 | 14 | 0 | 98 | |
| TRANSFORMERposition_encoding=relative2021.10 | 83.3 | — | 0 | 0 | 0 | — | — | |
| LEXLSTM2021.10 | 82.1 | — | 0 | 0 | 1.3 | — | — | |
| LSTM+LexPrediction paradigm=sequence prediction2021.12 | 82 | — | — | — | — | — | — | |
| LexLSTM2023.05 | 82 | — | — | 0 | 1 | 0 | 96 | |
| TransformerPrediction paradigm=sequence prediction2021.12 | 81 | — | — | — | — | — | — | |
| Universal Transformer - Attention DecoderPrediction paradigm=graph prediction2021.12 | 78.4 | — | — | — | — | — | — | |
| AM parserstructural trees=implicit or explicit use, conjunct order=disregarded or post-processed2023.05 | 78 | — | — | 100 | 97 | 72 | 76 | |
| Multiset Tagging and Latent Permutations - Lexlexicon mechanism=simple copy mechanism, conjunct order=disregarded or post-processed2023.05 | 77 | — | — | 73 | 92 | 35 | 79 | |
| Ours - LexLexicon Induction=false, Copy Mechanism=simple copy2023.05 | 75 | — | — | 71 | 82 | 5 | 79 | |
| TREE-MAML2021.10 | 66.7 | — | 0 | 0 | 0 | — | — | |
| Dynamic Least-to-MostEvaluation Protocol=Prompting2022.09 | — | 99.2 | — | — | — | — | — | |
| LeAREvaluation Protocol=Fully Supervised2022.09 | — | 97.7 | — | — | — | — | — | |
| LEAR2021.07 | — | 97.7 | — | — | — | — | — | |
| LEARabstraction=false2021.07 | — | 94.5 | — | — | — | — | — | |
| LEARsemantic locality=false2021.07 | — | 94 | — | — | — | — | — | |
| LEARTree-LSTM=false2021.07 | — | 80.7 | — | — | — | — | — | |
| LSTMarchitecture=Bidirectional2021.07 | — | 16 | — | — | — | — | — | |
| LSTMarchitecture=Unidirectional2021.07 | — | 32 | — | — | — | — | — | |
| T5-baseEvaluation Protocol=Fully Supervised2022.09 | — | 89.8 | — | — | — | — | — | |
| T5-base + CSLEvaluation Protocol=Fully Supervised2022.09 | — | 99.5 | — | — | — | — | — | |
| Transformer2021.07 | — | 35 | — | — | — | — | — |