Sentence Classification Transfer Tasks on SentEval
0.9209Average AccuracyPromptEOL
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| PromptEOLbackbone=LLaMA, Params=65B, fine-tuned=false2023.07 | 0.9209 | 0.9213 | 0.9343 | 0.9716 | 0.9191 | 0.9533 | 0.974 | 0.7728 | |
| PromptEOLbackbone=LLaMA, Params=30B, fine-tuned=false2023.07 | 0.9201 | 0.9164 | 0.9327 | 0.971 | 0.9186 | 0.9599 | 0.958 | 0.7843 | |
| PromptEOLbackbone=LLaMA, Params=13B, fine-tuned=false2023.07 | 0.9169 | 0.9202 | 0.9322 | 0.9729 | 0.914 | 0.9566 | 0.958 | 0.7646 | |
| ST5-EncParams=4.8B, fine-tuned=true2023.07 | 0.9163 | 0.9083 | 0.9444 | 0.9633 | 0.9168 | 0.9484 | 0.954 | 0.7791 | |
| PromptEOLbackbone=OPT, Params=66B, fine-tuned=false2023.07 | 0.9145 | 0.9096 | 0.934 | 0.9701 | 0.9193 | 0.9522 | 0.964 | 0.7525 | |
| AnglE-LLaMAbackbone=LLaMA2023.09 | 0.9138 | 0.9109 | 0.9354 | 0.9647 | 0.915 | 0.9528 | 0.964 | 0.7536 | |
| PromptEOLbackbone=OPT, Params=6.7B, fine-tuned=false2023.07 | 0.9134 | 0.9026 | 0.925 | 0.9667 | 0.9139 | 0.9467 | 0.96 | 0.7791 | |
| PromptEOLbackbone=OPT, Params=30B, fine-tuned=false2023.07 | 0.9125 | 0.9095 | 0.9277 | 0.9699 | 0.9179 | 0.9528 | 0.97 | 0.7397 | |
| PromptEOLbackbone=OPT, Params=13B, fine-tuned=false2023.07 | 0.9117 | 0.9073 | 0.929 | 0.9669 | 0.9148 | 0.9401 | 0.968 | 0.7559 | |
| PromptEOLbackbone=LLaMA, Params=7B, fine-tuned=false2023.07 | 0.9106 | 0.904 | 0.929 | 0.9688 | 0.9157 | 0.9511 | 0.954 | 0.7513 | |
| PromptEOLbackbone=OPT, Params=2.7B, fine-tuned=false2023.07 | 0.9077 | 0.8883 | 0.9229 | 0.9593 | 0.9176 | 0.9462 | 0.96 | 0.7594 | |
| ST5-EncParams=4.8B, fine-tuned=false2023.07 | 0.9071 | 0.9115 | 0.9333 | 0.9755 | 0.902 | 0.9407 | 0.944 | 0.7426 | |
| PromptEOLbackbone=OPT, Params=1.3B, fine-tuned=false2023.07 | 0.8987 | 0.8806 | 0.9155 | 0.959 | 0.9155 | 0.9308 | 0.95 | 0.7397 | |
| SimCSE-RoBERTaParams=220M, fine-tuned=true2023.07 | 0.8961 | 0.8112 | 0.9237 | 0.9511 | 0.9049 | 0.9275 | 0.918 | 0.7664 | |
| PromptRoBERTaParams=123M, fine-tuned=true2023.07 | 0.89 | 0.8574 | 0.9147 | 0.9481 | 0.9093 | 0.9253 | 0.904 | 0.771 | |
| SSCL-SimCSEPre-trained Encoder=BERT, Model Scale=large2023.05 | 0.8888 | 0.8601 | 0.9036 | 0.9598 | 0.8904 | 0.9127 | 0.932 | 0.7629 | |
| RankCSE (ListNet)Backbone=BERT_large2023.05 | 0.8878 | 0.8511 | 0.8956 | 0.9539 | 0.903 | 0.9077 | 0.932 | 0.7716 | |
| RankCSE (ListMLE)Backbone=BERT_large2023.05 | 0.8865 | 0.8463 | 0.8951 | 0.955 | 0.9008 | 0.9061 | 0.932 | 0.7699 | |
| ArcCSEBackbone=BERT_large2023.05 | 0.8854 | 0.8434 | 0.8882 | 0.9958 | 0.8979 | 0.905 | 0.92 | 0.7478 | |
| SimCSEBackbone=BERT_large2023.05 | 0.8834 | 0.8536 | 0.8938 | 0.9539 | 0.8963 | 0.9044 | 0.918 | 0.7641 | |
| RankCSE (ListNet)Backbone=RoBERTa_large2023.05 | 0.8812 | 0.8447 | 0.8951 | 0.9465 | 0.8987 | 0.8946 | 0.93 | 0.7588 | |
| SimCSE-RoBERTaParams=123M, fine-tuned=true2023.07 | 0.8808 | 0.8492 | 0.92 | 0.9411 | 0.8982 | 0.9127 | 0.888 | 0.7565 | |
| PCLBackbone=RoBERTa_large2023.05 | 0.8794 | 0.8447 | 0.8906 | 0.946 | 0.8926 | 0.8902 | 0.942 | 0.7496 | |
| L_c^metBackbone=BERT-base2024.02 | 0.8794 | — | — | — | — | — | — | — | |
| RankCSE (ListMLE)Backbone=RoBERTa_large2023.05 | 0.8787 | 0.8461 | 0.8927 | 0.9447 | 0.8999 | 0.8973 | 0.926 | 0.7443 | |
| L_c^mptBackbone=BERT-base2024.02 | 0.8756 | — | — | — | — | — | — | — | |
| SBERTbaseSupervision Type=Supervised, Backbone=BERT-base2022.03 | 0.8741 | 0.8364 | 0.8943 | 0.9439 | 0.8986 | 0.8896 | 0.896 | 0.76 | |
| PCLBackbone=BERT_large2023.05 | 0.8739 | 0.8247 | 0.8787 | 0.9504 | 0.8959 | 0.8775 | 0.93 | 0.76 | |
| RankCSE (ListNet)Backbone=RoBERTa_base2023.05 | 0.8735 | 0.8353 | 0.8922 | 0.9407 | 0.8897 | 0.8995 | 0.892 | 0.7652 | |
| RankCSE (ListNet)Backbone=BERT_base2023.05 | 0.8733 | 0.8321 | 0.8808 | 0.9525 | 0.9 | 0.8858 | 0.9 | 0.7617 | |
| RankCSE (ListMLE)Backbone=RoBERTa_base2023.05 | 0.8731 | 0.8332 | 0.8861 | 0.9403 | 0.8888 | 0.8907 | 0.908 | 0.7646 | |
| SimCSEPre-trained Encoder=BERT, Model Scale=large2023.05 | 0.8717 | 0.8437 | 0.8864 | 0.9526 | 0.8804 | 0.8995 | 0.904 | 0.7442 | |
| RankCSE (ListMLE)Backbone=BERT_base2023.05 | 0.8709 | 0.8307 | 0.8827 | 0.9506 | 0.899 | 0.877 | 0.894 | 0.7623 | |
| DiffCSEbackbone=RoBERTa2023.09 | 0.8704 | 0.8282 | 0.8861 | 0.9432 | 0.8771 | 0.8863 | 0.904 | 0.7681 | |
| SimCSEbackbone=RoBERTa2023.09 | 0.869 | 0.8337 | 0.8776 | 0.9505 | 0.8716 | 0.8902 | 0.908 | 0.7513 | |
| VisualCSE-RoBERTa_largeEvaluation Protocol=Linear Probing, Backbone=RoBERTa, Model Size=large, Learning Approach=Unsupervised2022.09 | 0.8686 | 0.8344 | 0.8813 | 0.9396 | 0.8843 | 0.8863 | 0.916 | 0.7386 | |
| L_arcBackbone=RoBERTa-base, Loss Function Type=Contrastive2024.02 | 0.8671 | 0.8306 | 0.8829 | 0.9423 | 0.8743 | 0.8861 | 0.8973 | 0.7563 | |
| AudioCSE-RoBERTa_largeEvaluation Protocol=Linear Probing, Backbone=RoBERTa, Model Size=large, Learning Approach=Unsupervised2022.09 | 0.8654 | 0.8349 | 0.8864 | 0.9386 | 0.8882 | 0.8863 | 0.924 | 0.6997 | |
| SimCSE-BERTbaseSupervision Type=Supervised, Backbone=BERT-base2022.03 | 0.8651 | 0.8269 | 0.8925 | 0.9481 | 0.8959 | 0.8731 | 0.884 | 0.7351 | |
| SimCSE-BERTbase + EHSupervision Type=Supervised, Backbone=BERT-base, Additional loss=Energy-based Hinge loss2022.03 | 0.8646 | 0.8281 | 0.8882 | 0.9434 | 0.8998 | 0.8814 | 0.862 | 0.749 | |
| VisualCSE-BERT_baseEvaluation Protocol=Linear Probing, Backbone=BERT, Model Size=base, Learning Approach=Unsupervised2022.09 | 0.863 | 0.8229 | 0.8718 | 0.9485 | 0.8934 | 0.8655 | 0.896 | 0.7432 | |
| L_m^MHEBackbone=RoBERTa-base, Modified Optimization Objective=true2024.02 | 0.8628 | 0.8262 | 0.8796 | 0.9373 | 0.8813 | 0.8819 | 0.8853 | 0.7478 | |
| L_m^MHSBackbone=RoBERTa-base, Modified Optimization Objective=true2024.02 | 0.8617 | 0.8268 | 0.8824 | 0.9356 | 0.8751 | 0.8722 | 0.8907 | 0.7492 | |
| ArcCSEBackbone=BERT_base2023.05 | 0.8612 | 0.7991 | 0.8525 | 0.9958 | 0.8921 | 0.849 | 0.892 | 0.7428 | |
| SimCSE-RoBERTa_largeEvaluation Protocol=Linear Probing, Backbone=RoBERTa, Model Size=large, Learning Approach=Unsupervised2022.09 | 0.8611 | 0.8274 | 0.8787 | 0.9366 | 0.8822 | 0.8858 | 0.92 | 0.6968 | |
| SimCSEBackbone=RoBERTa_large2023.05 | 0.8611 | 0.8274 | 0.8787 | 0.9366 | 0.8822 | 0.8858 | 0.92 | 0.6968 | |
| L_m^VBackbone=RoBERTa-base, Modified Optimization Objective=true2024.02 | 0.8611 | 0.8217 | 0.8762 | 0.9354 | 0.8771 | 0.8744 | 0.8887 | 0.7546 | |
| L_arcBackbone=BERT-base, Loss Function Type=Contrastive2024.02 | 0.8608 | 0.8161 | 0.8647 | 0.9496 | 0.8917 | 0.8594 | 0.8867 | 0.7573 | |
| DiffCSEBackbone=RoBERTa_base2023.05 | 0.8603 | 0.8242 | 0.8834 | 0.9351 | 0.8728 | 0.877 | 0.866 | 0.7635 | |
| L_m^BBackbone=RoBERTa-base, Modified Optimization Objective=true2024.02 | 0.8598 | 0.8204 | 0.8762 | 0.9363 | 0.8783 | 0.8775 | 0.8773 | 0.7528 | |
| PromCSE-BERTbase + EHSupervision Type=Supervised, Backbone=BERT-base, Additional loss=Energy-based Hinge loss2022.03 | 0.8591 | 0.818 | 0.8985 | 0.9392 | 0.9072 | 0.8705 | 0.826 | 0.7543 | |
| AudioCSE-BERT_baseEvaluation Protocol=Linear Probing, Backbone=BERT, Model Size=base, Learning Approach=Unsupervised2022.09 | 0.8587 | 0.811 | 0.8676 | 0.9465 | 0.8878 | 0.86 | 0.894 | 0.7443 | |
| DiffCSEBackbone=BERT_base2023.05 | 0.8587 | 0.8176 | 0.862 | 0.9476 | 0.8921 | 0.86 | 0.876 | 0.7554 | |
| IS-BERTbaseSupervision Type=Unsupervised, Backbone=BERT-base2022.03 | 0.8583 | 0.8109 | 0.8718 | 0.9496 | 0.8875 | 0.8596 | 0.8864 | 0.7424 | |
| SSCL-BERTPre-trained Encoder=BERT, Model Scale=large, Pooling=cls2023.05 | 0.8583 | 0.7393 | 0.8718 | 0.9496 | 0.8875 | 0.8596 | 0.8864 | 0.7424 | |
| ISBackbone=BERT_base2023.05 | 0.8583 | 0.8109 | 0.8718 | 0.9496 | 0.8875 | 0.8596 | 0.8864 | 0.7424 | |
| IS-BERT2023.09 | 0.8583 | 0.8109 | 0.8718 | 0.9496 | 0.8875 | 0.8596 | 0.8864 | 0.7424 | |
| SimCSE-BERT_baseEvaluation Protocol=Linear Probing, Backbone=BERT, Model Size=base, Learning Approach=Unsupervised2022.09 | 0.8582 | 0.8118 | 0.8646 | 0.9443 | 0.8887 | 0.855 | 0.898 | 0.7449 | |
| SimCSE-BERTbaseSupervision Type=Unsupervised, Backbone=BERT-base2022.03 | 0.8581 | 0.8118 | 0.8646 | 0.9445 | 0.8888 | 0.855 | 0.898 | 0.7443 | |
| SimCSEBackbone=BERT_base2023.05 | 0.8581 | 0.8118 | 0.8646 | 0.9445 | 0.8888 | 0.855 | 0.898 | 0.7443 | |
| L_infoBackbone=BERT-base, Loss Function Type=Contrastive2024.02 | 0.8581 | 0.8118 | 0.8646 | 0.9445 | 0.8888 | 0.855 | 0.898 | 0.7443 | |
| L_c^metBackbone=RoBERTa-base2024.02 | 0.8574 | — | — | — | — | — | — | — | |
| SSCL-SimCSEPre-trained Encoder=BERT, Model Scale=base2023.05 | 0.8561 | 0.8108 | 0.8616 | 0.9421 | 0.8863 | 0.8524 | 0.8961 | 0.742 | |
| InferSent-GloVeSupervision Type=Supervised, Backbone=GloVe2022.03 | 0.8559 | 0.8157 | 0.8654 | 0.925 | 0.9038 | 0.8418 | 0.882 | 0.7577 | |
| PromCSE-BERTbaseSupervision Type=Supervised, Backbone=BERT-base2022.03 | 0.855 | 0.8186 | 0.8856 | 0.9378 | 0.8969 | 0.8644 | 0.828 | 0.7536 | |
| PCLBackbone=RoBERTa_base2023.05 | 0.8549 | 0.8183 | 0.8755 | 0.9292 | 0.8721 | 0.8726 | 0.852 | 0.7646 | |
| PromCSE-BERTbaseSupervision Type=Unsupervised, Backbone=BERT-base2022.03 | 0.8546 | 0.8095 | 0.8546 | 0.945 | 0.8946 | 0.8484 | 0.884 | 0.7461 | |
| AudioCSE-RoBERTa_baseEvaluation Protocol=Linear Probing, Backbone=RoBERTa, Model Size=base, Learning Approach=Unsupervised2022.09 | 0.8542 | 0.8076 | 0.8718 | 0.9339 | 0.8729 | 0.8764 | 0.866 | 0.7507 | |
| L_m^MHSBackbone=BERT-base, Modified Optimization Objective=true2024.02 | 0.8542 | 0.804 | 0.8548 | 0.9431 | 0.8919 | 0.8545 | 0.8813 | 0.7497 | |
| L_m^VBackbone=BERT-base, Modified Optimization Objective=true2024.02 | 0.854 | 0.8082 | 0.8603 | 0.9429 | 0.8929 | 0.8527 | 0.8747 | 0.7467 | |
| PCLBackbone=BERT_base2023.05 | 0.8534 | 0.8011 | 0.8525 | 0.9422 | 0.8915 | 0.8512 | 0.874 | 0.7612 | |
| L_m^BBackbone=BERT-base, Modified Optimization Objective=true2024.02 | 0.8528 | 0.8069 | 0.8531 | 0.9435 | 0.8938 | 0.8534 | 0.8753 | 0.7434 | |
| VisualCSE-RoBERTa_baseEvaluation Protocol=Linear Probing, Backbone=RoBERTa, Model Size=base, Learning Approach=Unsupervised2022.09 | 0.8511 | 0.8184 | 0.876 | 0.9276 | 0.8714 | 0.8677 | 0.852 | 0.7443 | |
| SimCSEPre-trained Encoder=BERT, Model Scale=base2023.05 | 0.8511 | 0.8162 | 0.8544 | 0.9401 | 0.8805 | 0.8506 | 0.891 | 0.7403 | |
| Universal Sentence EncoderSupervision Type=Supervised2022.03 | 0.851 | 0.8009 | 0.8519 | 0.9398 | 0.867 | 0.8638 | 0.932 | 0.7014 | |
| L_c^mptBackbone=RoBERTa-base2024.02 | 0.851 | — | — | — | — | — | — | — | |
| L_m^MHEBackbone=BERT-base, Modified Optimization Objective=true2024.02 | 0.8507 | 0.8042 | 0.8531 | 0.941 | 0.891 | 0.8503 | 0.8673 | 0.7478 | |
| BERTbase (first-last avg.)Supervision Type=Unsupervised, Backbone=BERT-base, Pooling strategy=first-last average2022.03 | 0.8494 | 0.7866 | 0.8625 | 0.9437 | 0.8866 | 0.844 | 0.928 | 0.6954 | |
| last avg.Backbone=BERT_base2023.05 | 0.8494 | 0.7866 | 0.8625 | 0.9437 | 0.8866 | 0.844 | 0.928 | 0.6954 | |
| BERT avg.Params=110M, fine-tuned=false2023.07 | 0.8494 | 0.7866 | 0.8625 | 0.9437 | 0.8866 | 0.844 | 0.928 | 0.6954 | |
| BERTpooling=average2023.09 | 0.8494 | 0.7866 | 0.8625 | 0.9437 | 0.8866 | 0.844 | 0.928 | 0.6954 | |
| SimCSE-RoBERTa_baseEvaluation Protocol=Linear Probing, Backbone=RoBERTa, Model Size=base, Learning Approach=Unsupervised2022.09 | 0.8484 | 0.8104 | 0.8774 | 0.9328 | 0.8694 | 0.866 | 0.846 | 0.7368 | |
| SimCSEBackbone=RoBERTa_base2023.05 | 0.8484 | 0.8104 | 0.8774 | 0.9328 | 0.8694 | 0.866 | 0.846 | 0.7368 | |
| L_infoBackbone=RoBERTa-base, Loss Function Type=Contrastive2024.02 | 0.8484 | 0.8104 | 0.8774 | 0.9328 | 0.8694 | 0.866 | 0.846 | 0.7368 | |
| BERTbase (CLS)Supervision Type=Unsupervised, Backbone=BERT-base, Pooling strategy=CLS2022.03 | 0.8466 | 0.7868 | 0.8485 | 0.9421 | 0.8823 | 0.8413 | 0.914 | 0.7113 | |
| BERTPre-trained Encoder=BERT, Model Scale=large, Pooling=cls2023.05 | 0.8466 | 0.7868 | 0.8485 | 0.9421 | 0.8823 | 0.8413 | 0.914 | 0.7113 | |
| BERTpooling=CLS2023.09 | 0.8466 | 0.7868 | 0.8485 | 0.9421 | 0.8823 | 0.8413 | 0.914 | 0.7113 | |
| SSCL-BERTPre-trained Encoder=BERT, Model Scale=base, Pooling=cls2023.05 | 0.8455 | 0.8048 | 0.8588 | 0.9526 | 0.8697 | 0.8484 | 0.888 | 0.6962 | |
| SSCL-BERTPre-trained Encoder=BERT, Model Scale=base, Pooling=avg2023.05 | 0.8368 | 0.7887 | 0.8428 | 0.9531 | 0.874 | 0.8079 | 0.86 | 0.7312 | |
| Skip-thoughtSupervision Type=Unsupervised2022.03 | 0.835 | 0.765 | 0.801 | 0.936 | 0.871 | 0.82 | 0.922 | 0.73 | |
| Skip-thought2023.05 | 0.835 | 0.765 | 0.801 | 0.936 | 0.871 | 0.82 | 0.922 | 0.73 | |
| Skip-thoughtBackbone=Non-BERT2023.05 | 0.835 | 0.765 | 0.801 | 0.936 | 0.871 | 0.82 | 0.922 | 0.73 | |
| Skip-thoughtpooling=average2023.09 | 0.835 | 0.765 | 0.801 | 0.936 | 0.871 | 0.82 | 0.922 | 0.73 | |
| BERTPre-trained Encoder=BERT, Model Scale=base, Pooling=avg2023.05 | 0.8286 | 0.7767 | 0.8312 | 0.9446 | 0.8611 | 0.8008 | 0.8512 | 0.7264 | |
| BERTPre-trained Encoder=BERT, Model Scale=base, Pooling=cls2023.05 | 0.8257 | 0.7686 | 0.8268 | 0.9373 | 0.8587 | 0.8056 | 0.882 | 0.7013 | |
| GloVe embeddings (avg.)Supervision Type=Unsupervised, Backbone=GloVe, Pooling strategy=average2022.03 | 0.8152 | 0.7725 | 0.783 | 0.9117 | 0.8785 | 0.8018 | 0.83 | 0.7287 | |
| GloVe embeddingsPooling=average2023.05 | 0.8152 | 0.7725 | 0.783 | 0.9117 | 0.8785 | 0.8018 | 0.83 | 0.7287 | |
| GloVe (avg.)Backbone=Non-BERT2023.05 | 0.8152 | 0.7725 | 0.783 | 0.9117 | 0.8785 | 0.8018 | 0.83 | 0.7287 | |
| GloVepooling=average2023.09 | 0.8152 | 0.7725 | 0.783 | 0.9117 | 0.8785 | 0.8018 | 0.83 | 0.7287 |