Scientific Document Representation Evaluation on SCIDOCS (test)
87.1MAG F1Oracle SciDocs
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Oracle SciDocsupper_bound_protocol=triplets from SCIDOCS data2022.02 | 87.1 | 94.8 | 87.2 | 93.5 | 88.7 | 94.6 | 92.3 | 96.8 | 91.4 | 96.4 | 53.8 | 19.4 | 83 | |
| SPECTERsource=Cohan et al. (2020)2022.02 | 82 | 86.4 | 83.6 | 91.5 | 84.5 | 92.4 | 88.3 | 94.9 | 88.1 | 94.8 | 53.9 | 20 | 80 | |
| SciNCLtraining_data=Replicated SPECTER training data (w/ leakage), seeds=ten2022.02 | 81.4 | 88.7 | 85.3 | 92.3 | 87.5 | 93.9 | 93.6 | 97.3 | 91.6 | 96.4 | 53.9 | 19.3 | 81.8 | |
| SPECTERtraining_data=Random S2ORC training data (w/o leakage)2022.02 | 81.3 | 88.4 | 83.1 | 91.3 | 84 | 92.1 | 86.2 | 93.9 | 87.8 | 94.7 | 52.2 | 17.5 | 79.4 | |
| SciNCLtraining_data=Random S2ORC training data (w/o leakage)2022.02 | 81.3 | 89.4 | 84.3 | 91.8 | 85.6 | 92.8 | 91.4 | 96.3 | 90.1 | 95.7 | 54.3 | 19.9 | 81.1 | |
| DeCLUTRyear=20212022.02 | 81.2 | 88 | 63.4 | 80.6 | 60 | 78.6 | 57.2 | 77.4 | 62.9 | 80.9 | 52 | 17.4 | 66.6 | |
| USEyear=20182022.02 | 80 | 83.9 | 77.2 | 88.1 | 76.5 | 88.1 | 76.6 | 89 | 78.3 | 89.8 | 53.7 | 19.6 | 75.1 | |
| BERTyear=20192022.02 | 79.9 | 74.3 | 59.9 | 78.3 | 57.1 | 76.4 | 54.3 | 75.1 | 57.9 | 77.3 | 52.1 | 18.1 | 63.4 | |
| SciBERTsource=Cohan et al. (2020)2022.02 | 79.7 | 80.7 | 50.7 | 73.1 | 47.7 | 71.1 | 48.3 | 71.7 | 49.7 | 72.6 | 52.1 | 17.9 | 59.6 | |
| CiteBERTyear=20212022.02 | 78.8 | 74.8 | 53.2 | 73.6 | 49.9 | 71.3 | 45 | 67.9 | 50.3 | 72.1 | 51.6 | 17 | 58.8 | |
| BioBERTyear=20192022.02 | 77.2 | 73 | 53.3 | 74 | 50.6 | 72.2 | 45.5 | 69 | 49.4 | 71.8 | 52 | 17.9 | 58.8 | |
| SGCsource=Cohan et al. (2020)2022.02 | 76.8 | 82.7 | 77.2 | 88 | 75.7 | 87.5 | 91.6 | 96.2 | 84.1 | 92.5 | 52.7 | 18.2 | 76.9 | |
| Citeomaticsource=Cohan et al. (2020)2022.02 | 67.1 | 75.7 | 81.1 | 90.2 | 80.5 | 90.2 | 86.3 | 94.1 | 84.4 | 92.8 | 52.5 | 17.3 | 76 |