AMR Parsing on LDC2017T10 AMR 2.0 (test)
86.26SmatchGraphene Smatch
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Graphene Smatchensemble=SPRING(4), APT, T5, Cai&Lam, aggregation=Smatch2021.10 | 86.26 | 89.03 | 86.75 | 92.47 | 84.61 | 76.72 | 91.39 | 76.25 | 84.85 | — | — | — | — | — | |
| LeakDistillExtra Data=140K2023.06 | 86.1 | 88.8 | 86.5 | 91.6 | 83.9 | 76.6 | 91.4 | 75.1 | 82.4 | — | — | — | — | — | |
| Barzdins2021.10 | 85.93 | 88.85 | 86.63 | 92.51 | 84.01 | 75.73 | 91.16 | 76.53 | 84.64 | — | — | — | — | — | |
| Struct-BART + MBSE-A silver1+2+3 + ensemble dec.vocabulary=joint, silver=219K, distillation=Ensemble-5, ensemble-decoding=true2021.12 | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Graphene Supportensemble=SPRING(4), APT, T5, Cai&Lam, aggregation=Support2021.10 | 85.85 | 88.68 | 86.35 | 92.3 | 84.63 | 77.01 | 91.23 | 74.49 | 84.41 | — | — | — | — | — | |
| Struct-BART + MBSE-G silver1+2+3 + ensemble dec.vocabulary=joint, silver=219K, distillation=Ensemble-4, ensemble-decoding=true2021.12 | 85.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LeakDistillExtra Data=None2023.06 | 85.7 | 88.6 | 86.2 | 91.1 | 83.9 | 76.8 | 91 | 74.2 | 81.8 | — | — | — | — | — | |
| Struct-BART + MBSE-G silver1+2+3 + ensemble dec.vocabulary=separate, silver=219K, distillation=Ensemble-4, ensemble-decoding=true2021.12 | 85.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bai et al.silver=200K2021.12 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMRBART (large)Pre-trained=true, Model size=large2022.03 | 85.4 | 88.3 | 85.8 | 91.5 | 81.4 | 74 | 91.2 | 73.5 | 81.5 | — | — | — | — | — | |
| AMRBARTExtra Data=200K2023.06 | 85.4 | 88.3 | 85.8 | 91.5 | 81.4 | 74 | 91.2 | 73.5 | 81.5 | — | — | — | — | — | |
| ATPExtra Data=40K2023.06 | 85.2 | 88.3 | 85.6 | 93.1 | 83.3 | 74.9 | 90.7 | 74.7 | 83.3 | — | — | — | — | — | |
| StructBART-J ensem.Pre-trained Model=BART, Fine-tuning=true, Ensemble Decoding=true, Vocabulary Strategy=Joint2021.10 | 84.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Structured-BART-baseline + self-trained silver1 + ensemble dec.vocabulary=separate, silver=90K, ensemble-decoding=true2021.12 | 84.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AncestorExtra Data=None2023.06 | 84.8 | 88.1 | 85.3 | 91.8 | 84.1 | 74 | 90.5 | 75.1 | 83.4 | — | — | — | — | — | |
| Graphene 4Sensemble=4 SPRING checkpoints2021.10 | 84.78 | 87.96 | 85.29 | 92.19 | 83.88 | 75.22 | 90.64 | 71.42 | 83.46 | — | — | — | — | — | |
| StructBART-JPre-trained Model=BART, Fine-tuning=true, Extra Data=47K, Train Align.=true, Vocabulary Strategy=Joint2021.10 | 84.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Structured-BART-baseline + self-trained silver1vocabulary=separate, silver=90K2021.12 | 84.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BiBLExtra Data=None2023.06 | 84.6 | 87.8 | 85.1 | 92.5 | 83.6 | 73.9 | 90.3 | 74.4 | 83.1 | — | — | — | — | — | |
| Bevilacqua et al. (2021)Pre-trained Model=BART, Fine-tuning=true, Collapse Subgraph=G.R., External Dependency (Lemm.)=true2021.10 | 84.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al.silver=200K2021.12 | 84.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua+ (2021, large)Pre-trained=true, Model size=large2022.03 | 84.5 | 86.7 | 84.9 | 83.7 | 87.3 | 79.9 | 89.6 | 72.3 | 79.7 | — | — | — | — | — | |
| SPRING (ours)Extra Data=None2023.06 | 84.4 | 87.4 | 84.8 | 90.9 | 84.1 | 73.5 | 90.4 | 71.6 | 80.1 | — | — | — | — | — | |
| Bevilacqua et al.training_data=200K silver2021.04 | 84.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al. (2021)Pre-trained Model=BART, Fine-tuning=true, Extra Data=200K2021.10 | 84.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua+ (2021, large)Pre-trained=true, Model size=large, Silver data fine-tuning=true2022.03 | 84.3 | 86.7 | 84.8 | 90.5 | 83.1 | 73.6 | 90.8 | 72.4 | 80.5 | — | — | — | — | — | |
| SPRINGExtra Data=200K2023.06 | 84.3 | 86.7 | 84.8 | 90.5 | 83.1 | 73.6 | 90.8 | 72.4 | 80.5 | — | — | — | — | — | |
| SPRINGselection_criteria=highest Smatch among 4 random seeds2021.10 | 84.22 | 87.38 | 84.72 | 90.77 | 82.76 | 72.65 | 89.98 | 74.3 | 82.89 | — | — | — | — | — | |
| StructBART-JPre-trained Model=BART, Fine-tuning=true, Vocabulary Strategy=Joint2021.10 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Xia et al.silver=1.8M2021.12 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Structured-BART-baselinevocabulary=joint2021.12 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StructBART-SPre-trained Model=BART, Fine-tuning=true, Vocabulary Strategy=Separate2021.10 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Structured-BART-baselinevocabulary=separate2021.12 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al.2021.04 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al. (2021)Pre-trained Model=BART, Fine-tuning=true, External Dependency (Lemm.)=true2021.10 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMRBART (base)Pre-trained=true, Model size=base2022.03 | 83.6 | 86.7 | 84 | 90 | 78.6 | 73.7 | 90.2 | 71.3 | 79.5 | — | — | — | — | — | |
| APT basebackbone=RoBERTa, training_data=70K silver, decoding=partial ensemble2021.04 | 83.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T52021.10 | 82.98 | 86.17 | 83.43 | 90.65 | 77.99 | 73.43 | 89.85 | 72.44 | 82.02 | — | — | — | — | — | |
| APT basebackbone=RoBERTa, decoding=partial ensemble2021.04 | 82.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APT basebackbone=RoBERTa, training_data=70K silver2021.04 | 82.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APT2021.10 | 82.7 | 86.18 | 83.23 | 90.2 | 78.87 | 67.27 | 89.48 | 73.19 | 82.01 | — | — | — | — | — | |
| Bevilacqua+ (2021, base)Pre-trained=true, Model size=base2022.03 | 82.7 | 85.1 | 83.3 | 90 | 82.2 | 72 | 89.7 | 70.8 | 79.1 | — | — | — | — | — | |
| Zhou et al. (2021)Pre-trained Model=RoBERTa, Collapse Subgraph=S.A., External Dependency (NER)=true, Extra Data=70K, Train Align.=true2021.10 | 82.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhou et al.year=2021a, silver=70K2021.12 | 82.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APT smallbackbone=RoBERTa, decoding=partial ensemble2021.04 | 82.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APT basebackbone=RoBERTa2021.04 | 81.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APT smallbackbone=RoBERTa2021.04 | 81.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhou et al. (2021)Pre-trained Model=RoBERTa, Collapse Subgraph=S.A., External Dependency (NER)=true2021.10 | 81.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Xu et al. (2020)Pre-trained Model=Custom, Fine-tuning=true, Extra Data=4M2021.10 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Xu et al.silver=14M2021.12 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mining + synTxt + synAMRsynthetic data source=unlabeled text2020.10 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Lee et al.backbone=RoBERTa, training_data=85K silver2021.04 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Lee et al. (2020)Pre-trained Model=RoBERTa, Collapse Subgraph=S.A., External Dependency (NER)=true, Extra Data=85K, Train Align.=true2021.10 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Lee et al.silver=85K2021.12 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| synAMRsynthetic data source=unlabeled text2020.10 | 81 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| synTxt + synAMRsynthetic data source=unlabeled text2020.10 | 81 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mining + synAMRsynthetic data source=unlabeled text2020.10 | 80.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| synTxt2020.10 | 80.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mining + synTxt2020.10 | 80.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| oracle mining2020.10 | 80.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMR-gsGraph Re-categorization=true, BERT=true2020.04 | 80.2 | 82.8 | 80.8 | 81.1 | 86.3 | 78.9 | 88.1 | 64.6 | 74.2 | — | — | — | — | — | |
| Cai and LamGraph Recategorization=true, Embeddings=RoBERTa2020.10 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and LamBERT=base, Graph Recategorization=true2020.10 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| F. A. et al. + SE^RBackbone=stack-Transformer2020.10 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and Lambackbone=BERT, graph re-categorization=true2021.04 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Astudillo et al.backbone=RoBERTa2021.04 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Xu et al.training_data=4M silver2021.04 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and Lam (2020)Pre-trained Model=BERT, Collapse Subgraph=G.R.2021.10 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Astudillo et al. (2020)Pre-trained Model=RoBERTa, Collapse Subgraph=S.A., External Dependency (NER)=true2021.10 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and Lam2021.12 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fernandez Astudillo et al.2021.12 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CaiL (2020a)Pre-trained=true2022.03 | 80.2 | 82.8 | 80 | 81.1 | 86.3 | 78.9 | 88.1 | 64.6 | 74.2 | — | — | — | — | — | |
| Xu+ (2020)Pre-trained=true2022.03 | 80.2 | 83.7 | 80.8 | 85.4 | 75.1 | 71.5 | 87.4 | 66.5 | 78.9 | — | — | — | — | — | |
| Lee20Hand-crafted rules=true, Specialized pretraining=true2020.10 | 80.2 | — | — | — | — | — | 88.1 | — | 78.2 | — | — | — | — | — | |
| Cai20 w/ rulesHand-crafted rules=true, Specialized pretraining=true2020.10 | 80.2 | — | — | — | — | — | 88.1 | — | 74.2 | — | — | — | — | — | |
| Xu20Hand-crafted rules=false, Specialized pretraining=true2020.10 | 80.2 | — | — | — | — | — | 87.4 | — | 78.9 | — | — | — | — | — | |
| Cai&Lam2021.10 | 80.15 | 83.6 | 80.66 | 82.25 | 85.36 | 78.09 | 87.39 | 66.46 | 77.35 | — | — | — | — | — | |
| Stack-TransformerAttention Modification=Full, Variant=c2020.10 | 79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| F. A. et al.2020.10 | 79 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Stack-TransformerAttention Modification=buffer only, Variant=e2020.10 | 78.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMR-gsGraph Re-categorization=false, BERT=true2020.04 | 78.7 | 81.5 | 79.2 | 87.1 | 81.3 | 66.1 | 88.1 | 63.8 | 74.5 | — | — | — | — | — | |
| Cai and LamEmbeddings=BERT2020.10 | 78.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and LamBERT=base2020.10 | 78.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and Lambackbone=BERT2021.04 | 78.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai and Lam (2020)Pre-trained Model=BERT2021.10 | 78.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cai20 w/o rulesHand-crafted rules=false, Specialized pretraining=true2020.10 | 78.7 | — | — | — | — | — | 88.1 | — | 74.5 | — | — | — | — | — | |
| TransformerMulti-task learning=true, Variant=b2020.10 | 78 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhou+ (2020)Pre-trained=true2022.03 | 77.5 | 80.4 | 78.2 | 78.8 | 86.5 | 76.1 | 85.9 | 61.1 | 71 | — | — | — | — | — | |
| AMR-gsGraph Re-categorization=true, BERT=false2020.04 | 77.3 | 80.1 | 77.9 | 78.4 | 86.1 | 75.6 | 86.4 | 58.5 | 69.4 | — | — | — | — | — | |
| TransformerVariant=a2020.10 | 77.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| attention-based neural transducerbeam search=true2019.09 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhang et al.Graph Re-categorization=true, BERT=true2020.04 | 77 | 80 | 78 | 79 | 86 | 77 | 86 | 61 | 71 | — | — | — | — | — | |
| Zhang et al.Graph Recategorization=true, Embeddings=BERT2020.10 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhang et al.BERT=large, Graph Recategorization=true2020.10 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhang et al.backbone=BERT, graph re-categorization=true2021.04 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhang et al. (2019b)Pre-trained Model=BERT, Collapse Subgraph=G.R., External Dependency (POS)=true, External Dependency (NER)=true, External Dependency (Lemm.)=true2021.10 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhang et al.year=2019b2021.12 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zhang+ (2019b)Pre-trained=true2022.03 | 77 | 80 | 78 | 79 | 86 | 77 | 86 | 61 | 71 | — | — | — | — | — | |
| Zhang19Hand-crafted rules=true, Specialized pretraining=true2020.10 | 77 | — | — | — | — | — | 86 | — | 71 | — | — | — | — | — | |
| Lindemann20Hand-crafted rules=true, Specialized pretraining=true2020.10 | 76.8 | — | — | — | — | — | — | — | — | — | — | — | — | — |