Relation Extraction on NYT (test)
92.5F1 ScoreDIRECT
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| DIRECTCategory=Adjacency List (Fully)2021.06 | 92.5 | 92.3 | 92.8 | — | — | |
| DIRECTMatch Metric=Exact-Match2021.06 | 90.2 | 90.2 | 90.2 | — | — | |
| CASRELbackbone=BERT, initialization=pre-trained weights2019.09 | 89.6 | 89.7 | 89.5 | — | — | |
| CasRelCategory=Adjacency List (Partially)2021.06 | 89.6 | 89.7 | 89.5 | — | — | |
| CASREL_LSTMbackbone=LSTM, initialization=GloVe2019.09 | 83.6 | 84.2 | 83 | — | — | |
| WDecMatch Metric=Exact-Match2021.06 | 81.7 | 88.1 | 76.1 | — | — | |
| PNDecMatch Metric=Exact-Match2021.06 | 78.9 | 80.6 | 77.3 | — | — | |
| CASREL_randombackbone=BERT, initialization=random2019.09 | 78.5 | 81.5 | 75.7 | — | — | |
| ETL-Span2019.09 | 78 | 85.5 | 71.7 | — | — | |
| ETL-BIES2019.09 | 77.4 | 84.4 | 71.5 | — | — | |
| Seq2UMTreeMatch Metric=Exact-Match2021.06 | 77.1 | 79.1 | 75.1 | — | — | |
| OrderRLreported_in_other_works=†2019.09 | 72.1 | 77.9 | 67.2 | — | — | |
| CopyR_RL2019.09 | 72.1 | 77.9 | 67.2 | — | — | |
| CopyRLCategory=Edge List2021.06 | 72.1 | 77.9 | 67.2 | — | — | |
| CopyMTL_mulMatch Metric=Exact-Match2021.06 | 72 | 75.7 | 68.7 | — | — | |
| CopyR_RLre-implementation=true2019.09 | 71.1 | 72.8 | 69.4 | — | — | |
| CopyMTL_oneMatch Metric=Exact-Match2021.06 | 70.9 | 72.7 | 69.2 | — | — | |
| GraphRelreported_in_other_works=†2019.09 | 61.9 | 63.9 | 60 | — | — | |
| GraphRel_2ppasses=2p2019.09 | 61.9 | 63.9 | 60 | — | — | |
| GraphRel_2pCategory=Adjacency Matrices2021.06 | 61.9 | 63.9 | 60 | — | — | |
| GraphRel_1ppasses=1p2019.09 | 60 | 62.9 | 57.3 | — | — | |
| GraphRel_1pCategory=Adjacency Matrices2021.06 | 60 | 62.9 | 57.3 | — | — | |
| MultiHeadofficial_implementation=*2019.09 | 59.6 | 60.7 | 58.6 | — | — | |
| MHSMatch Metric=Exact-Match2021.06 | 59.6 | 60.7 | 58.6 | — | — | |
| MultiDecoderquoted_from_paper=‡2019.09 | 58.7 | 61 | 56.6 | — | — | |
| CopyR_Multi_Decoderdecoder_structure=multi-decoder2019.09 | 58.7 | 61 | 56.6 | — | — | |
| CopyRE_mulCategory=Edge List2021.06 | 58.7 | 61 | 56.6 | — | — | |
| CopyR_One_Decoderdecoder_structure=one-decoder2019.09 | 56 | 59.4 | 53.1 | — | — | |
| CopyRE_oneCategory=Edge List2021.06 | 56 | 59.4 | 53.1 | — | — | |
| DUALRE model=BiGRUs2020.11 | 54.55 | — | — | — | — | |
| DUALRE model=BERTs2020.11 | 53 | — | — | — | — | |
| DUALRE model=PaLSTMs2020.11 | 52.1 | — | — | — | — | |
| BAFixRE model=BERTs2020.11 | 51.45 | — | — | — | — | |
| BASetRE model=BERTs2020.11 | 50.81 | — | — | — | — | |
| DUALRE model=PCNNs2020.11 | 49.86 | — | — | — | — | |
| BASetRE model=BiGRUs2020.11 | 49.66 | — | — | — | — | |
| MaxThresRE model=BERTs2020.11 | 49.52 | — | — | — | — | |
| EntThresRE model=BERTs2020.11 | 49.4 | — | — | — | — | |
| DUALRE model=CNNs2020.11 | 47.44 | — | — | — | — | |
| BAFixRE model=BiLSTMs2020.11 | 47.07 | — | — | — | — | |
| MaxThresRE model=CNNs2020.11 | 46.55 | — | — | — | — | |
| CoTypeTime=4.1min2016.10 | 46.3 | 42.3 | 51.1 | — | — | |
| DS-OnlyRE model=BERTs2020.11 | 46.25 | — | — | — | — | |
| BAFixRE model=BiGRUs2020.11 | 46.13 | — | — | — | — | |
| BASetRE model=BiLSTMs2020.11 | 45.61 | — | — | — | — | |
| BASetRE model=PaLSTMs2020.11 | 45.55 | — | — | — | — | |
| EntThresRE model=BiGRUs2020.11 | 45.53 | — | — | — | — | |
| DS-OnlyRE model=BiGRUs2020.11 | 45.32 | — | — | — | — | |
| BAFixRE model=CNNs2020.11 | 45.32 | — | — | — | — | |
| MaxThresRE model=BiGRUs2020.11 | 45.31 | — | — | — | — | |
| DUALRE model=BiLSTMs2020.11 | 45.24 | — | — | — | — | |
| BAFixRE model=PaLSTMs2020.11 | 45.07 | — | — | — | — | |
| EntThresRE model=PaLSTMs2020.11 | 44.72 | — | — | — | — | |
| HA-OnlyRE model=PCNNs2020.11 | 44.63 | — | — | — | — | |
| DS-OnlyRE model=CNNs2020.11 | 44.63 | — | — | — | — | |
| MaxThresRE model=PaLSTMs2020.11 | 44.62 | — | — | — | — | |
| DS-OnlyRE model=PaLSTMs2020.11 | 44.29 | — | — | — | — | |
| EntThresRE model=CNNs2020.11 | 44.27 | — | — | — | — | |
| BASetRE model=CNNs2020.11 | 43.58 | — | — | — | — | |
| MaxThresRE model=BiLSTMs2020.11 | 43.5 | — | — | — | — | |
| DS-OnlyRE model=BiLSTMs2020.11 | 42.97 | — | — | — | — | |
| MaxThresRE model=PCNNs2020.11 | 42.58 | — | — | — | — | |
| EntThresRE model=BiLSTMs2020.11 | 42.1 | — | — | — | — | |
| NovelTagging2019.09 | 42 | 62.4 | 31.7 | — | — | |
| NovelTaggingCategory=Edge List2021.06 | 42 | 62.4 | 31.7 | — | — | |
| CoType-RMTime=2.6min2016.10 | 41.9 | 46.7 | 38 | — | — | |
| DS-OnlyRE model=PCNNs2020.11 | 41.77 | — | — | — | — | |
| EntThresRE model=PCNNs2020.11 | 41.54 | — | — | — | — | |
| CoType-TwoStepTime=9.6min2016.10 | 40.4 | 36.8 | 44.6 | — | — | |
| BAFixRE model=PCNNs2020.11 | 40.23 | — | — | — | — | |
| HA-OnlyRE model=CNNs2020.11 | 39.78 | — | — | — | — | |
| BASetRE model=PCNNs2020.11 | 35.84 | — | — | — | — | |
| DS-JointTime=22hr2016.10 | 35.4 | 57.4 | 25.6 | — | — | |
| MultiRTime=5.8min2016.10 | 33.3 | 33.8 | 32.7 | — | — | |
| LINETime=2.3min2016.10 | 33.2 | 33.5 | 32.9 | — | — | |
| NovelTaggingreported_in_other_works=†2019.09 | 31.7 | 32.8 | 30.6 | — | — | |
| DS+LogisticTime=25min2016.10 | 31.1 | 25.8 | 39.3 | — | — | |
| FCMTime=1.3hr2016.10 | 24 | 55.3 | 15.4 | — | — | |
| DeepWalkTime=1.1hr2016.10 | 19.7 | 17.6 | 22.4 | — | — | |
| HA-OnlyRE model=BERTs2020.11 | 19.53 | — | — | — | — | |
| DS+KernelTime=56hr2016.10 | 15.8 | 9.5 | 49 | — | — | |
| HA-OnlyRE model=BiLSTMs2020.11 | 15.04 | — | — | — | — | |
| HA-OnlyRE model=PaLSTMs2020.11 | 12.59 | — | — | — | — | |
| DS+PerceptronTime=15min2016.10 | 12.3 | 6.8 | 64.1 | — | — | |
| HA-OnlyRE model=BiGRUs2020.11 | 8.84 | — | — | — | — | |
| REBELModel Variant=Previous SotA, GPU=NVIDIA RTX 3090 (Train) / 4090 (Inference)2024.07 | — | — | — | 34 | 105 | |
| ReLiK RetrieverModel Variant=Retriever, GPU=NVIDIA RTX 3090 (Train) / 4090 (Inference)2024.07 | — | — | — | 2 | 2 | |
| ReLiK_BModel Variant=ReLiK_B (Reader), GPU=NVIDIA RTX 3090 (Train) / 4090 (Inference)2024.07 | — | — | — | 10 | 14 | |
| ReLiK_LModel Variant=ReLiK_L (Reader), GPU=NVIDIA RTX 3090 (Train) / 4090 (Inference)2024.07 | — | — | — | 23 | 28 | |
| ReLiK_SModel Variant=ReLiK_S (Reader), GPU=NVIDIA RTX 3090 (Train) / 4090 (Inference)2024.07 | — | — | — | 7 | 8 |