AMR parsing on AMR 3.0 (test)
86.5SMATCHBestgraph
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BestgraphTime (s)=512023.06 | 86.5 | — | 85.4 | 87.5 | 88 | 89 | 86.9 | 91.7 | 89.9 | 76.5 | 83.8 | 77.7 | |
| LeakDistillExtra Data=140K2023.06 | 84.6 | — | 80.9 | — | — | 87.5 | 84.9 | 90.7 | 87.8 | 73 | 81.3 | 73.4 | |
| LeakDistillExtra Data=None2023.06 | 84.5 | — | 80.7 | — | — | 87.5 | 84.9 | 90.5 | 88.5 | 73.7 | 80.7 | 73.1 | |
| Struct-BART + MBSE-A silver1+2+3 + ensemble dec.vocabulary=joint, silver=219K, distillation=Ensemble-5, ensemble-decoding=true2021.12 | 84.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Bai et al.silver=200K2021.12 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Struct-BART + MBSE-G silver1+2+3 + ensemble dec.vocabulary=joint, silver=219K, distillation=Ensemble-4, ensemble-decoding=true2021.12 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | |
| AMRBARTExtra Data=200K2023.06 | 84.2 | — | 80.3 | — | — | 87.1 | 84.6 | 90.2 | 88.5 | 72.1 | 78.9 | 72.4 | |
| Assemble! avgTime (s)=635, Corrupted graphs=222023.06 | 84.1 | — | 82.8 | 85.3 | 84.4 | 87.2 | 84.6 | 89.9 | 88.3 | 73.3 | 82.2 | 74.6 | |
| Struct-BART + MBSE-G silver1+2+3 + ensemble dec.vocabulary=separate, silver=219K, distillation=Ensemble-4, ensemble-decoding=true2021.12 | 84 | — | — | — | — | — | — | — | — | — | — | — | |
| Assemble! zeroTime (s)=256, Corrupted graphs=132023.06 | 83.9 | — | 82.5 | 85.1 | 85.4 | 87.1 | 84.4 | 89.9 | 88.3 | 74 | 82.2 | 74.3 | |
| BiBLExtra Data=None2023.06 | 83.9 | — | 81.9 | — | — | 87.2 | 84.3 | 89.8 | 93.2 | 68.1 | 83.7 | 73.8 | |
| ATPExtra Data=40K2023.06 | 83.9 | — | 82.5 | — | — | 87 | 84.3 | 89.7 | 88.4 | 73.9 | 81 | 73.9 | |
| Graphene SMATCHTime (s)=11,884, Corrupted graphs=2602023.06 | 83.8 | — | 82.6 | 85 | 85 | 86.9 | 84.4 | 89.9 | 88.1 | 73.8 | 81.3 | 73.7 | |
| Assemble!Block=Mergers, Time (s)=431, Corrupted graphs=62023.06 | 83.8 | — | 82.3 | 85 | 85.2 | 87 | 84.3 | 89.7 | 88.3 | 72.9 | 81.7 | 74.2 | |
| SPRINGExtra Data=None, Implementation=Authors' implementation2023.06 | 83.8 | — | 79.8 | — | — | 86.7 | 84.3 | 89.9 | 87.2 | 71.5 | 81.5 | 71.4 | |
| SMATCHavgTime (s)=493, Corrupted graphs=512023.06 | 83.7 | — | 82.4 | 85 | 85.3 | 86.8 | 84.2 | 89.7 | 88.1 | 73.3 | 82 | 73.9 | |
| GraphenebaseTime (s)=810, Corrupted graphs=3742023.06 | 83.6 | — | 82.4 | 84.8 | 84.9 | 86.6 | 84.1 | 89.8 | 88 | 73.5 | 81.2 | 72.3 | |
| AncestorExtra Data=None2023.06 | 83.5 | — | 82.2 | — | — | 86.6 | 84 | 89.5 | 88.9 | 72.6 | 81.5 | 74.2 | |
| StructBART-S ensem.Pre-trained Model=BART, Fine-tuning=true, Ensemble Decoding=true, Vocabulary Strategy=Separate2021.10 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | |
| SPRING1Time (s)=54, Corrupted graphs=512023.06 | 83.1 | — | 81.8 | 84.3 | 84.9 | 86.2 | 83.6 | 89.3 | 87.7 | 70.9 | 81.5 | 72.9 | |
| Bevilacqua et al.2021.04 | 83 | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al. (2021)Pre-trained Model=BART, Fine-tuning=true, External Dependency (Lemm.)=true2021.10 | 83 | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al. (2021)Pre-trained Model=BART, Fine-tuning=true, Extra Data=200K2021.10 | 83 | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al.silver=200K2021.12 | 83 | — | — | — | — | — | — | — | — | — | — | — | |
| SPRING3Time (s)=73, Corrupted graphs=212023.06 | 83 | — | 81.7 | 84.3 | 85.1 | 86.3 | 83.5 | 89.3 | 87.6 | 72.6 | 81.8 | 73.1 | |
| SPRINGExtra Data=None2023.06 | 83 | — | 79.1 | — | — | 85.4 | 83.5 | 89.5 | 87.1 | 71.7 | 81.2 | 71.3 | |
| SPRING4Time (s)=33, Corrupted graphs=322023.06 | 82.8 | — | 81.4 | 84 | 84.1 | 86 | 83.3 | 88.9 | 87.3 | 71.7 | 81.5 | 72.8 | |
| StructBART-SPre-trained Model=BART, Fine-tuning=true, Extra Data=47K, Train Align.=true, Vocabulary Strategy=Separate2021.10 | 82.7 | — | — | — | — | — | — | — | — | — | — | — | |
| SPRING2Time (s)=52, Corrupted graphs=322023.06 | 82.7 | — | 81.4 | 83.9 | 82.1 | 85.9 | 83.2 | 89 | 87.5 | 72.6 | 80.2 | 73 | |
| StructBART-JPre-trained Model=BART, Fine-tuning=true, Extra Data=47K, Train Align.=true, Vocabulary Strategy=Joint2021.10 | 82.6 | — | — | — | — | — | — | — | — | — | — | — | |
| SPRING5Time (s)=104, Corrupted graphs=3262023.06 | 82.6 | — | 81.4 | 83.9 | 84.5 | 85.8 | 83.2 | 89.2 | 87.3 | 73 | 81.6 | 73 | |
| StructBART-SPre-trained Model=BART, Fine-tuning=true, Vocabulary Strategy=Separate2021.10 | 82.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Structured-BART-baselinevocabulary=separate2021.12 | 82.3 | — | — | — | — | — | — | — | — | — | — | — | |
| StructBART-JPre-trained Model=BART, Fine-tuning=true, Vocabulary Strategy=Joint2021.10 | 82 | — | — | — | — | — | — | — | — | — | — | — | |
| Structured-BART-baselinevocabulary=joint2021.12 | 82 | — | — | — | — | — | — | — | — | — | — | — | |
| APT basebackbone=RoBERTa, decoding=partial ensemble2021.04 | 81.2 | — | — | — | — | — | — | — | — | — | — | — | |
| APT basebackbone=RoBERTa2021.04 | 80.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Zhou et al. (2021)Pre-trained Model=RoBERTa, Collapse Subgraph=S.A., External Dependency (NER)=true2021.10 | 80.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Zhou et al.year=2021a, silver=70K2021.12 | 80.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Bevilacqua et al. (2021)Pre-trained Model=BART, Fine-tuning=true, Collapse Subgraph=G.R., External Dependency (Lemm.)=true2021.10 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Lyu et al.2021.04 | 75.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Lyu et al.2021.12 | 75.8 | — | — | — | — | — | — | — | — | — | — | — | |
| latent segmentation (rule)use_hand_crafted_rules=true2020.10 | 75.8 | 88 | 72.6 | — | — | — | — | — | — | — | — | — | |
| latent segmentation (full)mode=full2020.10 | 75.6 | 87.8 | 72.9 | — | — | — | — | — | — | — | — | — | |
| latent segmentation (greedy)decoding=greedy2020.10 | 74.8 | 87 | 71.5 | — | — | — | — | — | — | — | — | — |