Named Entity Recognition on CoNLL English 2003 (test)
94.6F1 ScoreACE
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ACEFine-tuning=true, Trained on both train and development set=true2020.10 | 94.6 | — | — | — | — | |
| Yamada et al. (2020)Document features=yes2020.11 | 94.3 | — | — | — | — | |
| Yamada et al.2020.10 | 94.3 | — | — | — | — | |
| XLM-RFine-tuning=true, Trained on both train and development set=true2020.10 | 94.1 | — | — | — | — | |
| Transformer-Linear (+DEV)Approach=Fine-tuning (Ablations), Document features=yes (+enforce), Training=Includes dev data2020.11 | 94.09 | — | — | — | — | |
| Transformer-LinearApproach=Fine-tuning (Ablations), Document features=yes (+enforce)2020.11 | 93.75 | — | — | — | — | |
| Transformer-LinearApproach=Fine-tuning, Document features=yes2020.11 | 93.64 | — | — | — | — | |
| Baevski et al.model_year=20192019.11 | 93.5 | — | — | — | — | |
| Yu et al. (2020)Document features=yes2020.11 | 93.5 | — | — | — | — | |
| Baevski et al.2020.10 | 93.5 | — | — | — | — | |
| Yu et al.2020.10 | 93.5 | — | — | — | — | |
| Yu et al. (2020)2021.05 | 93.5 | — | — | 93.7 | 93.3 | |
| Straková et al.2020.10 | 93.4 | — | — | — | — | |
| LSTM-CRFembeddings=ELMo + BERT + Flair2019.08 | 93.38 | — | — | — | — | |
| Straková et al. (2019)Document features=yes2020.11 | 93.38 | — | — | — | — | |
| Hire-NER + BERTLanguage Models=BERT, Tagging=BIOES2019.11 | 93.37 | — | — | — | — | |
| BERT-MRC+DSCLoss=Self-adjusting Dice Loss (DSC)2019.11 | 93.33 | — | — | 93.41 | 93.25 | |
| Liu et al. 2019b (BERT)Language Models=BERT, Strict BIO converting=true2019.11 | 93.23 | — | — | — | — | |
| LSTM-CRFembeddings=BERT + Flair2019.08 | 93.22 | — | — | — | — | |
| Akbik, Blythe, and Vollgrafmodel_year=20182019.11 | 93.2 | — | — | — | — | |
| Akbik, Bergmann, and Vollgraf 2019Language Models=true, Trained on both training and development datasets=true2019.11 | 93.18 | — | — | — | — | |
| Akbik et al. (2019b)Document features=pooling2020.11 | 93.18 | — | — | — | — | |
| BERT-MRC+DLLoss=Dice Loss2019.11 | 93.17 | — | — | 93.22 | 93.12 | |
| LSTM-CRF (all layer mean)Approach=Feature-based, Document features=yes2020.11 | 93.12 | — | — | — | — | |
| BERT-MRC+FLLoss=Focal Loss2019.11 | 93.11 | — | — | 93.13 | 93.09 | |
| Flair2019.08 | 93.09 | — | — | — | — | |
| Akbik, Blythe, and Vollgraf 2018Language Models=true, Trained on both training and development datasets=true2019.11 | 93.09 | — | — | — | — | |
| seq2seqembeddings=ELMo + BERT + Flair2019.08 | 93.07 | — | — | — | — | |
| BERT-MRC2019.11 | 93.04 | — | — | 92.33 | 94.61 | |
| BERT-MRC2019.10 | 93.04 | — | — | 92.33 | 94.61 | |
| seq2seqembeddings=BERT + Flair2019.08 | 93 | — | — | — | — | |
| seq2seqembeddings=BERT + ELMo2019.08 | 92.99 | — | — | — | — | |
| seq2seqembeddings=BERT2019.08 | 92.98 | — | — | — | — | |
| LSTM-CRFembeddings=BERT2019.08 | 92.94 | — | — | — | — | |
| Locate and Label2021.05 | 92.94 | — | — | 92.13 | 93.73 | |
| LSTM-CRFembeddings=BERT + ELMo2019.08 | 92.93 | — | — | — | — | |
| BERT2019.08 | 92.8 | — | — | — | — | |
| Devlin et al. 2019 (BERT)Language Models=BERT2019.11 | 92.8 | — | — | — | — | |
| BERT-Tagger2019.11 | 92.8 | — | — | — | — | |
| Devlin et al. (2019)2021.05 | 92.8 | — | — | — | — | |
| BERT-Tagger2019.10 | 92.8 | — | — | — | — | |
| Transformer-LinearApproach=Fine-tuning, Document features=no2020.11 | 92.79 | — | — | — | — | |
| FlairVersion=v0.72021.01 | 92.7 | — | — | — | — | |
| Clark et al. 2018Language Models=true2019.11 | 92.61 | — | — | — | — | |
| Devlin et al.model_year=20182019.11 | 92.6 | — | — | — | — | |
| CVT2019.11 | 92.6 | — | — | — | — | |
| CVT2019.10 | 92.6 | — | — | — | — | |
| LSTM-CRFembeddings=ELMo2019.08 | 92.58 | — | — | — | — | |
| seq2seqembeddings=ELMo2019.08 | 92.43 | — | — | — | — | |
| Neural-CRF+AEtraining_data=training and development sets, evaluation_protocol=average over 5 runs2018.08 | 92.29 | — | — | — | — | |
| LSTM-CRFembeddings=Flair2019.08 | 92.25 | — | — | — | — | |
| Peters et al. (2018)+ELMoEmbeddings=ELMo2018.08 | 92.22 | — | — | — | — | |
| ELMo2019.08 | 92.22 | — | — | — | — | |
| ELMo2019.11 | 92.22 | — | — | — | — | |
| Peters et al. (2018)2021.05 | 92.22 | — | — | — | — | |
| ELMO2019.10 | 92.22 | — | — | — | — | |
| Peters et al. 2018 (ELMo)Language Models=ELMo2019.11 | 92.2 | — | — | — | — | |
| Flair-MLtraining=multilingual2019.11 | 92.2 | — | — | — | — | |
| Trankit2021.01 | 92.1 | — | — | — | — | |
| StanzaVersion=v1.1.12021.01 | 92.1 | — | — | — | — | |
| Hire-NERTagging=BIOES2019.11 | 91.96 | — | — | — | — | |
| Neural-CRF+AEtraining_data=training set only, evaluation_protocol=average over 5 runs, statistically_significant=true2018.08 | 91.89 | — | — | — | — | |
| seq2seqembeddings=Flair2019.08 | 91.87 | — | — | — | — | |
| LSTM-CRF (all layer mean)Approach=Feature-based, Document features=no2020.11 | 91.83 | — | — | — | — | |
| Qian et al. 20192019.11 | 91.74 | — | — | — | — | |
| Liu et al. 2018External knowledge=true2019.11 | 91.71 | — | — | — | — | |
| LSTM-CRF+char-IntNet-5Character Embedding Model=IntNet-52018.10 | 91.64 | — | — | — | — | |
| Xin et al. 20182019.11 | 91.64 | — | — | — | — | |
| Yang et al. (2017)2018.08 | 91.62 | — | — | — | — | |
| Chiu and Nichols (2016)training_data=training and development sets2018.08 | 91.62 | — | — | — | — | |
| Yang, Zhang, and Dong 20172019.11 | 91.62 | — | — | — | — | |
| Chiu and Nichols 2016External knowledge=true, Trained on both training and development datasets=true2019.11 | 91.62 | — | — | — | — | |
| Zhang, Liu, and Song 20182019.11 | 91.57 | — | — | — | — | |
| Liu et al. 2019bStrict BIO converting=true2019.11 | 91.54 | — | — | — | — | |
| Chen et al. 20192019.11 | 91.44 | — | — | — | — | |
| LSTM-CRF+char-IntNet-9Character Embedding Model=IntNet-92018.10 | 91.39 | — | — | — | — | |
| Yang and Zhang 20182019.11 | 91.35 | — | — | — | — | |
| BERT-MLtraining=multi-task learning2019.11 | 91.3 | — | — | — | — | |
| Liu et al. 20182019.11 | 91.24 | — | — | — | — | |
| Ma and Hovy (2016)2018.08 | 91.21 | — | — | — | — | |
| LSTM-CRF+char-CNNCharacter Embedding Model=CNN, Reference=Ma and Hovy, 20162018.10 | 91.21 | — | — | — | — | |
| Ma and Hovy 20162019.11 | 91.21 | — | — | — | — | |
| CNN-BiLSTM-CRF (Ma and Hovy, 2016)2019.11 | 91.21 | — | — | — | — | |
| Ma and Hovy (2016)Pos tags information=Not used2019.06 | 91.21 | — | — | — | — | |
| Luo et al. (2015)2018.08 | 91.2 | — | — | — | — | |
| GRM-CRF+char-GRUCharacter Embedding Model=GRU, Reference=Yang et al., 20172018.10 | 91.2 | — | — | — | — | |
| BERT-SLtraining=single language2019.11 | 91.2 | — | — | — | — | |
| LSTM-CRF+char-LSTMCharacter Embedding Model=LSTM, Type=Re-implementation2018.10 | 91.13 | — | — | — | — | |
| LSTM-CRF+char-CNNCharacter Embedding Model=CNN, Type=Re-implementation2018.10 | 91.11 | — | — | — | — | |
| BiLSTM-BiLSTM-CRF (Akhundov et al., 2018)2019.11 | 91.11 | — | — | — | — | |
| Liu et al. 2019a2019.11 | 91.1 | — | — | — | — | |
| CN32019.11 | 91.1 | — | — | — | — | |
| Neural-CRFtraining_data=training set only, evaluation_protocol=average over 5 runs2018.08 | 91.06 | — | — | — | — | |
| BiLSTM-CRF2019.10 | 91.03 | — | — | — | — | |
| Lample et al. (2016)2018.08 | 90.94 | — | — | — | — | |
| LSTM-CRF+char-LSTMCharacter Embedding Model=LSTM, Reference=Lample et al., 20162018.10 | 90.94 | — | — | — | — | |
| Lample et al.2019.08 | 90.94 | — | — | — | — | |
| Lample et al. 20162019.11 | 90.94 | — | — | — | — | |
| BiLSTM-BiLSTM-CRF (Lample et al., 2016)2019.11 | 90.94 | — | — | — | — | |
| Lample et al. (2016)Pos tags information=Not used2019.06 | 90.94 | — | — | — | — |