Named Entity Recognition on BC5CDR (test)
94.73Macro F1 (span-level)BioBERT-v1.1
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| BioBERT-v1.1Number of Parameters=> 60M2022.09 | 94.73 | 94.47 | 95 | |
| DistilBioBERTNumber of Parameters=> 60M2022.09 | 94.53 | 94.04 | 95.04 | |
| CompactBioBERTNumber of Parameters=> 60M2022.09 | 94.31 | 94.03 | 94.6 | |
| BioMegatronNumber of Parameters=345m, Vocabulary=Bio-vocab-30k, Fine-tuning=30 epochs, Pre-training=From scratch on PubMed2020.10 | 92.9 | 92.1 | 93.6 | |
| PubMedBERTNumber of Parameters=110m, Vocabulary=PubMedBERT-vocab (30k), Fine-tuning=30 epochs2020.10 | 92.6 | 92.1 | 93.2 | |
| BioMegatronNumber of Parameters=345m, Vocabulary=Bio-vocab-50k, Fine-tuning=30 epochs, Pre-training=From scratch on PubMed2020.10 | 92.5 | 92.9 | 92 | |
| DistilBERTNumber of Parameters=> 60M2022.09 | 92.5 | 92.11 | 92.9 | |
| TinyBioBERTNumber of Parameters=15M2022.09 | 92.2 | 91.31 | 93.09 | |
| BioMegatronNumber of Parameters=800m, Vocabulary=BERT-cased, Fine-tuning=30 epochs, Pre-training=From scratch on PubMed2020.10 | 92.1 | 91.3 | 92.9 | |
| BINDERSupervision=Fully Supervised2022.08 | 91.9 | 92.6 | 91.2 | |
| BioBERTNumber of Parameters=110m, Vocabulary=BERT-cased, Fine-tuning=30 epochs2020.10 | 91.7 | 90 | 93.4 | |
| BioMegatronNumber of Parameters=1.2b, Vocabulary=BERT-uncased, Fine-tuning=30 epochs, Pre-training=Fine-tuned from general domain model2020.10 | 91.3 | 92 | 90.5 | |
| Wang et al.Supervision=Fully Supervised2022.08 | 90.9 | — | — | |
| ELECTRAMedAveraging=5 runs with different seeds2021.04 | 90.03 | 88.76 | 91.34 | |
| SciBERTtraining_mode=finetune2019.03 | 90.01 | — | — | |
| SCIBERTEvaluation Protocol=Finetune2019.03 | 90.01 | — | — | |
| RL+DS+PA2021.04 | 89.93 | 92.05 | 87.91 | |
| Nooralahzadeh et al.Supervision=Fully Supervised2022.08 | 89.9 | 92.1 | 87.9 | |
| Spark NLP2021.04 | 89.73 | — | — | |
| BioFLAIR2021.04 | 89.42 | — | — | |
| BioBERTtraining_mode=finetune2019.03 | 88.85 | — | — | |
| SOTA (Li et al., 2016)2019.03 | 88.85 | — | — | |
| SCIBERTEvaluation Protocol=Frozen2019.03 | 88.73 | — | — | |
| BioMegatronNumber of Parameters=345m, Vocabulary=Bio-vocab-50k, Fine-tuning=30 epochs, Pre-training=From scratch on PubMed2020.10 | 88.5 | 86.1 | 91 | |
| BioMegatronNumber of Parameters=800m, Vocabulary=BERT-cased, Fine-tuning=30 epochs, Pre-training=From scratch on PubMed2020.10 | 87.9 | 85.8 | 90.1 | |
| HGN (BioBERT) (DOT)extra resources=false, backbone=BioBERT, aggregation=DOT2022.05 | 87.86 | 86.27 | 89.51 | |
| HGN (BioBERT) (MLP)extra resources=false, backbone=BioBERT, aggregation=MLP2022.05 | 87.77 | 86.7 | 88.86 | |
| HGN (BioBERT) (CONCAT)extra resources=false, backbone=BioBERT, aggregation=CONCAT2022.05 | 87.33 | 85.9 | 88.81 | |
| PubMedBERTNumber of Parameters=110m, Vocabulary=PubMedBERT-uncased (30k), Fine-tuning=30 epochs2020.10 | 87.3 | 86.2 | 88.4 | |
| HGN (BioBERT) (ADD)extra resources=false, backbone=BioBERT, aggregation=ADD2022.05 | 87.29 | 85.89 | 88.74 | |
| BioBERTNumber of Parameters=110m, Vocabulary=BERT-cased, Fine-tuning=30 epochs2020.10 | 87.2 | 85 | 89.4 | |
| BioBERTextra resources=false2022.05 | 87.15 | 86.47 | 87.84 | |
| BioMegatronNumber of Parameters=345m, Vocabulary=Bio-vocab-30k, Fine-tuning=30 epochs, Pre-training=From scratch on PubMed2020.10 | 87 | 85.2 | 88.8 | |
| BERT-BaseEvaluation Protocol=Finetune2019.03 | 86.72 | — | — | |
| BioBERT-v1.1Number of Parameters=> 60M2022.09 | 86.67 | 85.81 | 87.54 | |
| NCBI_BERTextra resources=false2022.05 | 86.6 | — | — | |
| BioMegatronNumber of Parameters=1.2b, Vocabulary=BERT-uncased, Fine-tuning=30 epochs, Pre-training=Fine-tuned from general domain model2020.10 | 86.4 | 83.8 | 89.2 | |
| KEBIO-LMextra resources=true2022.05 | 86.1 | — | — | |
| DistilBioBERTNumber of Parameters=> 60M2022.09 | 85.42 | 84.34 | 86.54 | |
| CompactBioBERTNumber of Parameters=> 60M2022.09 | 85.38 | 84.76 | 86.01 | |
| BERT-BaseEvaluation Protocol=Frozen2019.03 | 85.08 | — | — | |
| DistilBERTNumber of Parameters=> 60M2022.09 | 82.01 | 81.57 | 82.47 | |
| BINDERSupervision=Distantly Supervised2022.08 | 81.6 | 87.6 | 76.3 | |
| TinyBioBERTNumber of Parameters=15M2022.09 | 81.28 | 79.91 | 82.71 | |
| BONDDictionary=Full2022.10 | 81.1 | 76.6 | 86.1 | |
| RoSTERDictionary=Full2022.10 | 80.7 | 78.6 | 83 | |
| Conf-MPUSupervision=Distantly Supervised2022.08 | 80.1 | 76.6 | 83.8 | |
| AutoNERSupervision=Distantly Supervised2022.08 | 80 | 82.6 | 77.5 | |
| StandardDictionary=Full2022.10 | 79.7 | 82.8 | 76.8 | |
| HighGEN + RoSTERDictionary=Pseudo2022.10 | 74.6 | 73.3 | 76 | |
| Unified-00Supervision=no-supervision, Training Source=Other Biomedical Datasets2019.09 | 73.8 | 84.1 | 65.7 | |
| BERT-ESSupervision=Distantly Supervised2022.08 | 73.7 | 80.4 | 67.9 | |
| HighGEN + BONDDictionary=Pseudo2022.10 | 72.9 | 69.5 | 76.7 | |
| HighGEN (class)shot=5-shot (per entity type), level=class-level2022.10 | 72.5 | — | — | |
| HighGEN + StandardDictionary=Pseudo2022.10 | 72.2 | 77.9 | 67.3 | |
| HighGEN + RoSTERDictionary=Pseudo, Ablation=w/o L2022.10 | 72.2 | 68.3 | 76.6 | |
| GeNER + RoSTERDictionary=Pseudo2022.10 | 72.1 | 74.6 | 69.7 | |
| Unified-11Supervision=no-supervision, Training Source=Other Biomedical Datasets2019.09 | 70.2 | 73.8 | 67 | |
| GeNER + BONDDictionary=Pseudo2022.10 | 69.3 | 69 | 69.7 | |
| HighGEN (entity)shot=5-shot (per entity type), level=entity-level2022.10 | 68.2 | — | — | |
| TALLOR2021.07 | 66.73 | 66.53 | 66.94 | |
| QUIPshot=5-shot (per entity type)2022.10 | 65.7 | — | — | |
| GeNER + StandardDictionary=Pseudo2022.10 | 64.9 | 76.6 | 56.3 | |
| Dict/KB MatchingSupervision=Distantly Supervised2022.08 | 64.3 | 86.4 | 51.2 | |
| MTM-VoteSupervision=no-supervision, Training Source=Other Biomedical Datasets2019.09 | 63.6 | 64.4 | 62.8 | |
| Entity-oriented demonstrationMode=fixed, Strategy=search, Template=context, Number of training instances=50, Backbone=bert-base-cased2021.10 | 62.87 | — | — | |
| Ours w/o Instance SelectionInstance Selection=false2021.07 | 60.95 | 58.7 | 63.37 | |
| BNPUSupervision=Distantly Supervised2022.08 | 59.2 | 48.1 | 77.1 | |
| Supervisedshot=5-shot (per entity type)2022.10 | 55 | — | — | |
| BERT+CRF w/o demonstrationNumber of training instances=25, Backbone=bert-base-cased2021.10 | 52.56 | — | — | |
| Ours w/o AutophraseAutophrase=false2021.07 | 45.68 | 74.56 | 32.93 | |
| Unified-01Supervision=no-supervision, Training Source=Other Biomedical Datasets2019.09 | 42.7 | 93.7 | 27.6 | |
| Self-training2021.07 | 42.19 | 73.69 | 29.55 | |
| AutoNER2021.07 | 35.52 | 42.22 | 30.66 | |
| Seed Rules + Neural TaggerIteration Learning=false2021.07 | 33.86 | 78.33 | 21.6 | |
| CGExpanMax Rules=10002021.07 | 30.86 | 40.96 | 24.75 | |
| Our Learned Rules2021.07 | 29.94 | 79.29 | 18.46 | |
| HMM-agg.2021.07 | 29 | 43.7 | 21.6 | |
| LinkedHMM2021.07 | 12.32 | 10.18 | 15.6 | |
| Seed Rules2021.07 | 7.33 | 94.09 | 3.81 |