Relation Classification on SemEval-2010 Task 8 (test)
91.9F1 ScoreSP question, BERT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SP question, BERTVariant=question, Backbone=BERT-large, Supervised=true2020.10 | 91.9 | — | 90.7 | 93.2 | |
| LiTianBackbone=BERT, Supervised=true2020.10 | 91 | — | 94.2 | 88 | |
| SP token, BERTVariant=token, Backbone=BERT-large, Supervised=true2020.10 | 90.7 | — | 92.8 | 88.8 | |
| R-BERTbase model=BERT-Base-Uncased2019.05 | 89.25 | — | — | — | |
| RC MTB, BERTBackbone=BERT-large, Supervised=true2020.10 | 89.2 | — | — | — | |
| Att-Pooling-CNN2019.05 | 88 | — | — | — | |
| SP relation, BERTVariant=relation, Backbone=BERT-large, Supervised=true2020.10 | 87.1 | — | 91.9 | 83.1 | |
| DRNNsFeatures=Word+POS+GR+WordNet embeddings + data augmentation2016.01 | 86.1 | — | — | — | |
| Attention CNN2019.05 | 85.9 | — | — | — | |
| depLCNN+NSFeature Sets=+WordNet, words around nominals2015.06 | 85.6 | — | — | — | |
| depLCNNFeatures=Word + WordNet + words around nominals + negative sampling from NYT dataset2016.01 | 85.6 | — | — | — | |
| Entity Attention Bi-LSTM2019.05 | 85.2 | — | — | — | |
| C-GCNEvaluation Protocol=with-mention2018.09 | 84.8 | — | — | — | |
| SPTreeEvaluation Protocol=with-mention, Implementation=Reported in original paper2018.09 | 84.4 | — | — | — | |
| DRNNsFeatures=Word+POS+GR+WordNet embeddings w/o data augmentation2016.01 | 84.2 | — | — | — | |
| CR-CNNFeature Set=word embeddings, word position embeddings2015.04 | 84.1 | — | — | — | |
| CR-CNNFeature set=Word embeddings, position embeddings, Special treatment for Other class=true2015.08 | 84.1 | — | — | — | |
| CR-CNN (ranking-loss)Features=word embedding2015.05 | 84.1 | — | — | — | |
| CR-CNNspecial treatment to Other class=true2015.11 | 84.1 | — | — | — | |
| VOTE-BIDIRECTstrategy=hybrid-voting2015.11 | 84.1 | — | — | — | |
| VOTE-BACKWARDstrategy=hybrid-voting2015.11 | 84.1 | — | — | — | |
| CR-CNNFeatures=Word embeddings + position embeddings2016.01 | 84.1 | — | — | — | |
| Ensemble MethodsFeatures=Word+POS+NER+WordNet embeddings, CNNs, RNNs + Voting2016.01 | 84.1 | — | — | — | |
| CR-CNN2019.05 | 84.1 | — | — | — | |
| depLCNN+NSFeature Sets=None2015.06 | 84 | — | — | — | |
| depLCNNFeature Sets=+WordNet, words around nominals2015.06 | 83.7 | — | — | — | |
| SDP-LSTMFeature set=Word embeddings, POS embeddings, WordNet embeddings, grammar relation embeddings2015.08 | 83.7 | — | — | — | |
| SDP-LSTM2015.11 | 83.7 | — | — | — | |
| SDP-LSTMFeatures=Word + POS + GR + WordNet embeddings2016.01 | 83.7 | — | — | — | |
| depLCNNFeatures=Word + WordNet + words around nominals2016.01 | 83.7 | — | — | — | |
| SDP-LSTMEvaluation Protocol=with-mention, Implementation=Reported in original paper2018.09 | 83.7 | — | — | — | |
| DepNNAdditional Features=NER, Use of AF=true2015.07 | 83.6 | — | — | — | |
| DepNN + linearFeatures=word embedding, dependency paths, WordNet, NER2015.05 | 83.6 | — | — | — | |
| DepNN2015.11 | 83.6 | — | — | — | |
| DepNNFeatures=Word embeddings + NER2016.01 | 83.6 | — | — | — | |
| RelEmb (task-spec embedding) + linearFeatures=word embedding, dependency paths, WordNet, NE2015.05 | 83.5 | — | — | — | |
| CNNarchitecture=convolutional neural network baseline2015.11 | 83.5 | — | — | — | |
| (5) FCM (log-linear) + linear (Hybrid)Features=word embedding, dependency parse, NER2015.05 | 83.4 | — | — | — | |
| Hybrid FCM2015.11 | 83.4 | — | — | — | |
| STACK-FORWARDstrategy=hybrid-stacking2015.11 | 83.4 | — | — | — | |
| Ensemble MethodsFeatures=Word+POS+NER+WordNet embeddings, CNNs, RNNs + Stacking2016.01 | 83.4 | — | — | — | |
| (5) FCM (log-linear) + linear (Hybrid)Features=word embedding, dependency parse, WordNet2015.05 | 83.1 | — | — | — | |
| FCMFeature Set=word embeddings, dependency parse, NER2015.04 | 83 | — | — | — | |
| FCMAdditional Features=NER, Use of AF=true2015.07 | 83 | — | — | — | |
| FCMFeature set=Word embeddings, dependency parsing, NER2015.08 | 83 | — | — | — | |
| (2) FCM (log-bilinear)Features=word embedding, dependency parse, NER2015.05 | 83 | — | — | — | |
| FCM2015.11 | 83 | — | — | — | |
| FCMFeatures=Word embeddings + dependency parsing, NER2016.01 | 83 | — | — | — | |
| DepNNFeatures=Word embeddings + WordNet2016.01 | 83 | — | — | — | |
| FCM2019.05 | 83 | — | — | — | |
| CR-CNNFeature Set=word embeddings2015.04 | 82.8 | — | — | — | |
| DepNNUse of AF=false2015.07 | 82.8 | — | — | — | |
| CR-CNNFeature set=Word embeddings, Special treatment for Other class=true2015.08 | 82.8 | — | — | — | |
| RelEmb (task-spec embedding)Features=word embedding2015.05 | 82.8 | — | — | — | |
| DepNNFeatures=word embedding, dependency paths2015.05 | 82.8 | — | — | — | |
| CR-CNNFeatures=Word embeddings2016.01 | 82.8 | — | — | — | |
| CNN+SoftmaxFeature Set=word embeddings, word position embeddings, word pair, words around word pair, WordNet2015.04 | 82.7 | — | — | — | |
| CNNFeature Sets=+WordNet, words around nominals2015.06 | 82.7 | — | — | — | |
| CNNAdditional Features=WordNet, Use of AF=true2015.07 | 82.7 | — | — | — | |
| CNNFeature set=Word embeddings, word position embeddings, WordNet2015.08 | 82.7 | — | — | — | |
| Chain CNNFeature set=Word embeddings, POS, NER, WordNet2015.08 | 82.7 | — | — | — | |
| CR-CNNFeature set=Word embeddings, position embeddings, Special treatment for Other class=false2015.08 | 82.7 | — | — | — | |
| CNN (Zeng et al., 2014)Features=word embedding, WordNet2015.05 | 82.7 | — | — | — | |
| CR-CNN (log-loss)Features=word embedding2015.05 | 82.7 | — | — | — | |
| CNNReference=Zeng et al., 20142015.11 | 82.7 | — | — | — | |
| CNNFeatures=Word embeddings + position embeddings, WordNet2016.01 | 82.7 | — | — | — | |
| Chain CNNFeatures=Word embeddings, POS, NER, WordNet2016.01 | 82.7 | — | — | — | |
| PA-LSTMEvaluation Protocol=with-mention, Implementation=Produced using open implementation2018.09 | 82.7 | — | — | — | |
| CNN+Softmax2019.05 | 82.7 | — | — | — | |
| (2) FCM (log-bilinear)Features=word embedding, dependency parse, WordNet2015.05 | 82.5 | — | — | — | |
| MVRNNFeature Set=word embeddings, POS, NER, WordNet2015.04 | 82.4 | — | — | — | |
| MVRNNFeature Sets=+POS, NER, WordNet2015.06 | 82.4 | — | — | — | |
| MVRNNFeature set=Word embeddings, POS, NER, WordNet2015.08 | 82.4 | — | — | — | |
| CR-CNNFeature set=Word embeddings, Special treatment for Other class=false2015.08 | 82.4 | — | — | — | |
| MVRNN + linearFeatures=word embedding, syntactic parse, POS, NER, WordNet2015.05 | 82.4 | — | — | — | |
| MVRNN2015.11 | 82.4 | — | — | — | |
| BACKWARDarchitecture=backward RNN baseline2015.11 | 82.4 | — | — | — | |
| MVRNNFeatures=Word embeddings + POS, NER, WordNet2016.01 | 82.4 | — | — | — | |
| SDP-LSTMFeatures=Word embeddings2016.01 | 82.4 | — | — | — | |
| MVRNN2019.05 | 82.4 | — | — | — | |
| SVMFeature Set=POS, prefixes, morphological, WordNet, dependency parse, Levin classes, ProBank, FrameNet, NomLex-Plus, Google n-gram, paraphrases, TextRunner2015.04 | 82.2 | — | — | — | |
| SVMFeature Sets=16 types of features2015.06 | 82.2 | — | — | — | |
| SVMAdditional Features=POS, prefixes, PropBank, Google n-gram, NomLex-Plus, Levin classes, WordNet, dependency parse, morphological, FrameNet, TextRunner, paraphrases, Use of AF=true2015.07 | 82.2 | — | — | — | |
| FCMUse of AF=false2015.07 | 82.2 | — | — | — | |
| SVMFeature set=POS, WordNet, prefixes and other morphological features, dependency parse, Levin classes, PropBank, FrameNet, NomLex-Plus, Google n-gram, paraphrases, TextRunner2015.08 | 82.2 | — | — | — | |
| SVM (Rink and Harabagiu, 2010)Features=POS, prefixes, morphological, WordNet, dependency parse, Levin classed, ProBank, FrameNet, NomLex-Plus, Google n-gram, paraphrases, TextRunner2015.05 | 82.2 | — | — | — | |
| SVM2015.11 | 82.2 | — | — | — | |
| SVMFeatures=POS, WordNet, prefixes and other morphological features, depdency parse, Levin classes, PropBank, FanmeNet, NomLex-Plus, Google n-gram, paraphrases, TextRunner2016.01 | 82.2 | — | — | — | |
| SVMEvaluation Protocol=with-mention, Implementation=Reported in original paper2018.09 | 82.2 | — | — | — | |
| SVM2019.05 | 82.2 | — | — | — | |
| (1) FCM (log-linear)Features=word embedding, dependency parse, WordNet2015.05 | 82 | — | — | — | |
| depLCNNFeature Sets=None2015.06 | 81.9 | — | — | — | |
| FORWARDarchitecture=forward RNN baseline2015.11 | 81.9 | — | — | — | |
| MV-RNNAdditional Features=POS, NER, WordNet, Use of AF=true2015.07 | 81.8 | — | — | — | |
| RelEmb (word2vec embedding)Features=word embedding2015.05 | 81.8 | — | — | — | |
| BIDIRECTarchitecture=bidirectional RNN baseline2015.11 | 81.8 | — | — | — | |
| (1) FCM (log-linear)Features=word embedding, dependency parse, NER2015.05 | 81.4 | — | — | — | |
| depCNNFeature Sets=None2015.06 | 81.3 | — | — | — | |
| FCMFeature Set=word embeddings2015.04 | 80.6 | — | — | — | |
| FCMFeature set=Word embeddings2015.08 | 80.6 | — | — | — |