Named Entity Recognition on OntoNotes 5.0 (test)
91.74F1 ScoreBaseline + BS
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Baseline + BSboundary smoothing=true2022.04 | 91.74 | 91.75 | 91.74 | — | — | — | — | — | — | — | |
| BERT-Biaffine ModelEncoder=BERT2021.08 | 91.3 | — | — | — | — | — | — | — | — | — | |
| Yu et al.2022.04 | 91.3 | 91.1 | 91.5 | — | — | — | — | — | — | — | |
| Baseline2022.04 | 91.21 | 90.31 | 92.13 | — | — | — | — | — | — | — | |
| Weighted Sampling DistributionEncoder=BERT2021.08 | 91.17 | — | — | — | — | — | — | — | — | — | |
| BERT-MRCEncoder=BERT2021.08 | 91.11 | — | — | — | — | — | — | — | — | — | |
| Li et al.source=2020b2022.04 | 91.11 | 92.98 | 89.95 | — | — | — | — | — | — | — | |
| Seq2Seqparadigm=generative, calibration=E-NER2023.05 | 90.64 | — | — | — | — | — | — | 0.0328 | — | — | |
| Vanilla Negative SamplingEncoder=BERT2021.08 | 90.59 | — | — | — | — | — | — | — | — | — | |
| W2NERMethod category=Seq2Seq2021.12 | 90.5 | 90.03 | 90.97 | — | — | — | — | — | — | — | |
| Hire-NERExternal Knowledge / Language Models=BERT2019.11 | 90.3 | — | — | — | — | — | — | — | — | — | |
| HCR w/ BERTEncoder=BERT2021.08 | 90.3 | — | — | — | — | — | — | — | — | — | |
| Yan et al.Method category=Seq2Seq, re-implementation=true2021.12 | 90.27 | 89.62 | 90.92 | — | — | — | — | — | — | — | |
| Seq2Seqparadigm=generative, calibration=EDL2023.05 | 90.22 | — | — | — | — | — | — | 0.0329 | — | — | |
| Yu et al.Method category=Span-based, re-implementation=true2021.12 | 89.89 | 90.01 | 89.77 | — | — | — | — | — | — | — | |
| Seq2Seqparadigm=generative, calibration=softmax2023.05 | 89.89 | — | — | — | — | — | — | 0.0375 | — | — | |
| Akbik, Bergmann, and VollgrafExternal Knowledge / Language Models=Language Models (Flair)2019.11 | 89.71 | — | — | — | — | — | — | — | — | — | |
| Akbik, Blythe, and VollgrafTraining setting=Single-language, Evaluation protocol=Supervised2019.11 | 89.7 | — | — | — | — | — | — | — | — | — | |
| Flair EmbeddingEncoder=Flair2021.08 | 89.3 | — | — | — | — | — | — | — | — | — | |
| Clark et al.External Knowledge / Language Models=Language Models2019.11 | 88.88 | — | — | — | — | — | — | — | — | — | |
| Clark et al.Training=Semi-supervised and multi-task learning2019.08 | 88.81 | — | — | — | — | — | — | — | — | — | |
| Clark et al.Training setting=Single-language, Evaluation protocol=Supervised2019.11 | 88.8 | — | — | — | — | — | — | — | — | — | |
| BERT-Taggerparadigm=sequence labeling, calibration=E-NER2023.05 | 88.74 | — | — | — | — | — | — | 0.0603 | — | — | |
| SpanNERparadigm=span-based, calibration=E-NER2023.05 | 88.44 | — | — | — | — | — | — | 0.0434 | — | — | |
| Att-BiLSTM-CNN2019.08 | 88.4 | 88.71 | 88.11 | — | — | — | — | — | — | — | |
| BERT-MLTraining setting=Multi-language, Evaluation protocol=Supervised2019.11 | 88.3 | — | — | — | — | — | — | — | — | — | |
| Cross-BiLSTM-CNN2019.08 | 88.27 | 88.37 | 88.17 | — | — | — | — | — | — | — | |
| BERT-Taggerparadigm=sequence labeling, calibration=softmax2023.05 | 88.2 | — | — | — | — | — | — | 0.1053 | — | — | |
| BiLSTM-LANBackbone=BiLSTM, Decoder=LAN2019.08 | 88.16 | — | — | — | — | — | — | — | — | — | |
| BERT-Taggerparadigm=sequence labeling, calibration=EDL2023.05 | 88.09 | — | — | — | — | — | — | 0.0838 | — | — | |
| Ghaddar and LanglaisTraining setting=Single-language, Evaluation protocol=Supervised2019.11 | 88 | — | — | — | — | — | — | — | — | — | |
| Hire-NERExternal Knowledge / Language Models=None2019.11 | 87.98 | — | — | — | — | — | — | — | — | — | |
| Ghaddar and LanglaisFeatures=Lexical features2019.08 | 87.95 | — | — | — | — | — | — | — | — | — | |
| Ghaddar and LanglaisExternal Knowledge / Language Models=External Lexicon (Wikipedia)2019.11 | 87.95 | — | — | — | — | — | — | — | — | — | |
| BERT-SLTraining setting=Single-language, Evaluation protocol=Supervised2019.11 | 87.9 | — | — | — | — | — | — | — | — | — | |
| BERT-SLEng 0-shotTraining setting=Single-language (English), Evaluation protocol=Supervised (Source Language)2019.11 | 87.9 | — | — | — | — | — | — | — | — | — | |
| SpanNERparadigm=span-based, calibration=softmax2023.05 | 87.82 | — | — | — | — | — | — | 0.0609 | — | — | |
| Baseline-BiLSTM-CNN2019.08 | 87.75 | 88.37 | 87.14 | — | — | — | — | — | — | — | |
| Chen et al.External Knowledge / Language Models=None2019.11 | 87.67 | — | — | — | — | — | — | — | — | — | |
| Ghaddar and LanglaisExternal Knowledge / Language Models=None2019.11 | 87.44 | — | — | — | — | — | — | — | — | — | |
| SpanNERparadigm=span-based, calibration=EDL2023.05 | 87.39 | — | — | — | — | — | — | 0.0474 | — | — | |
| BiLSTM-CRFBackbone=BiLSTM, Decoder=CRF2019.08 | 86.99 | — | — | — | — | — | — | — | — | — | |
| CRF-BiLSTM2019.08 | 86.99 | — | — | — | — | — | — | — | — | — | |
| Strubell et al.Architecture=Iterated dilated convolutions2019.08 | 86.84 | — | — | — | — | — | — | — | — | — | |
| Strubell et al.External Knowledge / Language Models=None2019.11 | 86.84 | — | — | — | — | — | — | — | — | — | |
| CRF-IDCNN2019.08 | 86.84 | — | — | — | — | — | — | — | — | — | |
| Strubell et al.Method category=Sequence Labeling2021.12 | 86.84 | — | — | — | — | — | — | — | — | — | |
| Shen et al.External Knowledge / Language Models=None2019.11 | 86.63 | — | — | — | — | — | — | — | — | — | |
| Shen et al.Learning Strategy=Active learning2019.08 | 86.52 | — | — | — | — | — | — | — | — | — | |
| BLSTM-CNN + emb + lexArchitecture=BLSTM-CNN, Features=Collobert word embeddings + lexicon (SENNA and DBpedia)2015.11 | 86.28 | 86.04 | 86.53 | — | — | — | — | — | — | — | |
| Chiu and NicholsBackbone=BiLSTM, Encoding=CNN character encoding2019.08 | 86.28 | — | — | — | — | — | — | — | — | — | |
| Chiu and NicholsExternal Knowledge / Language Models=None2019.11 | 86.28 | — | — | — | — | — | — | — | — | — | |
| BiLSTM-CNN2019.08 | 86.28 | 86.04 | 86.53 | — | — | — | — | — | — | — | |
| Chiu and Nichols2022.04 | 86.28 | 86.04 | 86.53 | — | — | — | — | — | — | — | |
| BLSTM-CNN + embArchitecture=BLSTM-CNN, Features=Collobert word embeddings2015.11 | 86.17 | 85.99 | 86.36 | — | — | — | — | — | — | — | |
| Durrett and Klein (2014)Architecture=Non-neural model2015.11 | 84.04 | 85.22 | 82.89 | — | — | — | — | — | — | — | |
| Durrett and Kleinyear=20142017.02 | 84.04 | — | — | — | — | — | — | — | — | — | |
| Durrett and Klein2019.08 | 84.04 | — | — | — | — | — | — | — | — | — | |
| Durrett and KleinExternal Knowledge / Language Models=None2019.11 | 84.04 | — | — | — | — | — | — | — | — | — | |
| BiLSTM-softmaxBackbone=BiLSTM, Decoder=softmax2019.08 | 83.76 | — | — | — | — | — | — | — | — | — | |
| Ratinov and Roth (2009)Architecture=Non-neural model2015.11 | 83.45 | 82 | 84.95 | — | — | — | — | — | — | — | |
| Ratinov and Rothyear=20092017.02 | 83.45 | — | — | — | — | — | — | — | — | — | |
| BLSTM-CNNArchitecture=BLSTM-CNN2015.11 | 82.53 | 82.58 | 82.49 | — | — | — | — | — | — | — | |
| Finkel and Manning (2009)Architecture=Non-neural model2015.11 | 82.42 | 84.04 | 80.86 | — | — | — | — | — | — | — | |
| Pradhan et al.Training setting=Single-language, Evaluation protocol=Supervised2019.11 | 82.4 | — | — | — | — | — | — | — | — | — | |
| Passos et al. (2014)Architecture=Non-neural model2015.11 | 82.24 | — | — | — | — | — | — | — | — | — | |
| BLSTMArchitecture=Bidirectional LSTM2015.11 | 77.77 | 79.68 | 75.97 | — | — | — | — | — | — | — | |
| FFNN + emb + caps + lexArchitecture=Feed-forward Neural Network, Features=embeddings, capitalization, lexicon2015.11 | 73.94 | 74.28 | 73.61 | — | — | — | — | — | — | — | |
| instance-oriented demonstrationStrategy=SBERT, Template=lexical, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 63.34 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=search, Template=context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.63 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=popular, Template=context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.59 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=random, Template=context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.58 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=search, Template=lexical, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.51 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=popular, Template=lexical, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.5 | — | — | — | — | — | — | — | — | — | |
| instance-oriented demonstrationStrategy=BERTScore, Template=context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.46 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=random, Template=lexical, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.41 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=search, Template=no-context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.38 | — | — | — | — | — | — | — | — | — | |
| instance-oriented demonstrationStrategy=SBERT, Template=context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.33 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=popular, Template=no-context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.31 | — | — | — | — | — | — | — | — | — | |
| entity-oriented demonstrationStrategy=random, Template=no-context, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.28 | — | — | — | — | — | — | — | — | — | |
| instance-oriented demonstrationStrategy=BERTScore, Template=lexical, Backbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 62.26 | — | — | — | — | — | — | — | — | — | |
| BERT+CRF w/o demonstrationBackbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 61.22 | — | — | — | — | — | — | — | — | — | |
| BERT-ML2 0-shotTraining setting=Multi-language (2 languages), Evaluation protocol=Supervised (Source Languages)2019.11 | 61 | — | — | — | — | — | — | — | — | — | |
| Entity-oriented demonstrationMode=fixed, Strategy=search, Template=context, Number of training instances=50, Backbone=bert-base-cased2021.10 | 59 | — | — | — | — | — | — | — | — | — | |
| BERT+CRF w/o demonstrationNumber of training instances=50, Backbone=bert-base-cased2021.10 | 54.51 | — | — | — | — | — | — | — | — | — | |
| NNShotBackbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 46.67 | — | — | — | — | — | — | — | — | — | |
| StructShotBackbone=bert-base-cased, Source Dataset=CoNLL03, Evaluation Protocol=Label Sharing2021.10 | 43.61 | — | — | — | — | — | — | — | — | — | |
| Instance-oriented demonstrationMode=variable, Strategy=SBERT, Template=context, Number of training instances=25, Backbone=bert-base-cased2021.10 | 42.18 | — | — | — | — | — | — | — | — | — | |
| BERT+CRF w/o demonstrationNumber of training instances=25, Backbone=bert-base-cased2021.10 | 38.97 | — | — | — | — | — | — | — | — | — | |
| Instance-oriented demonstrationMode=variable, Strategy=SBERT, Template=lexical, Number of training instances=25, Backbone=bert-base-cased2021.10 | 36.58 | — | — | — | — | — | — | — | — | — | |
| AEiOBackbone=GPT-3.5-Turbo-0125, Prompting Strategy=monolithic prompting, Number of entity types=182026.06 | — | — | — | — | — | — | — | — | 56.88 | 43.12 | |
| Bert-basebackbone=BERT-base, shot-size=52022.03 | — | — | — | — | 61.1 | — | — | — | — | — | |
| BERT-MRC-DSC2022.09 | — | — | — | — | — | — | 92.1 | — | — | — | |
| Bi-LSTMcontext=sentence2017.02 | — | — | — | 24.44 | — | — | — | — | — | — | |
| Bi-LSTM-CRFcontext=sentence2017.02 | — | — | — | 1 | — | — | — | — | — | — | |
| Bi-LSTM-CRF-Doccontext=document2017.02 | — | — | — | 1.32 | — | — | — | — | — | — | |
| CFNERIncremental Setting=FG-1-PG-12022.10 | — | — | — | — | 58.94 | 42.22 | — | — | — | — | |
| CFNERIncremental Setting=FG-2-PG-22022.10 | — | — | — | — | 72.59 | 55.96 | — | — | — | — | |
| CFNERIncremental Setting=FG-8-PG-12022.10 | — | — | — | — | 78.92 | 57.51 | — | — | — | — | |
| CFNERIncremental Setting=FG-8-PG-22022.10 | — | — | — | — | 80.68 | 60.52 | — | — | — | — |