Semantic Textual Similarity on STS 2012-2016 (test)
81.08STS-12 ScoreSimCSE + MixSP
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| SimCSE + MixSPBase Encoder=SimCSE, Backbone=BERT-Base, MixSP=true2024.06 | 81.08 | 89.74 | 90.41 | 91.27 | 84.16 | 85.67 | 87.18 | 75.86 | |
| DiffAug + MixSPBase Encoder=DiffAug, Backbone=BERT-Base, MixSP=true2024.06 | 80.88 | 88.89 | 89.64 | 91.46 | 84.53 | 85.47 | 87 | 75.88 | |
| SBERT + MoEBase Encoder=SBERT, Backbone=BERT-Base, MoE=true2024.06 | 80.71 | 87.89 | 89.88 | 90.83 | 83.75 | 84.89 | 86.14 | 75.01 | |
| DiffAug + MoEBase Encoder=DiffAug, Backbone=BERT-Base, MoE=true2024.06 | 80.18 | 88.74 | 90.25 | 91.48 | 83.63 | 85.25 | 86.29 | 76.19 | |
| SBERT + MixSPBase Encoder=SBERT, Backbone=BERT-Base, MixSP=true2024.06 | 79.99 | 89.47 | 89.99 | 90.89 | 84.59 | 85.49 | 87.27 | 76.23 | |
| SimCSE + MoEBase Encoder=SimCSE, Backbone=BERT-Base, MoE=true2024.06 | 79.38 | 89.02 | 89.8 | 91.35 | 83.38 | 85.1 | 86.34 | 76.43 | |
| SimCSE + FTBase Encoder=SimCSE, Backbone=BERT-Base, Fine-tuned=true2024.06 | 79.15 | 89.44 | 89.33 | 90.46 | 83.5 | 85.39 | 87.04 | 80.4 | |
| AnglE-LLaMA2-7Blearning_paradigm=Supervised, Backbone=LLaMA2-7B, fine-tuning=LoRA, prompt=Summarize sentence {sentence} in one word:2023.09 | 79 | 90.56 | 85.79 | 89.43 | 87 | 85.96 | 88.97 | 80.94 | |
| SimCSE-LLaMA2-7Blearning_paradigm=Supervised, Backbone=LLaMA2-7B, fine-tuning=LoRA2023.09 | 78.39 | 89.95 | 84.8 | 88.5 | 86.04 | 85.24 | 87.86 | 81.11 | |
| SimCSETraining Setting=Transfer from NLI2022.01 | 77.5 | 87.3 | 82.4 | 86.7 | 83.9 | 83.6 | — | — | |
| DiffAug + FTBase Encoder=DiffAug, Backbone=BERT-Base, Fine-tuned=true2024.06 | 77.3 | 89.56 | 87.77 | 88.54 | 82.02 | 84.21 | 85.36 | 78.94 | |
| DiffAugBase Encoder=DiffAug, Backbone=BERT-Base2024.06 | 76.92 | 85.17 | 80.81 | 86.91 | 82.52 | 82.42 | 84.32 | 80.27 | |
| SimCSEBase Encoder=SimCSE, Backbone=BERT-Base2024.06 | 75.3 | 84.67 | 80.19 | 85.4 | 80.82 | 81.57 | 84.25 | 80.39 | |
| SimCSE-BERTlearning_paradigm=Supervised, Backbone=BERT2023.09 | 75.3 | 84.67 | 80.19 | 85.4 | 80.82 | 81.57 | 84.25 | 80.39 | |
| AnglE-BERTlearning_paradigm=Supervised, Backbone=BERT2023.09 | 75.09 | 85.56 | 80.66 | 86.44 | 82.47 | 82.37 | 85.16 | 81.23 | |
| ConSERT-BERTlearning_paradigm=Supervised, Backbone=BERT2023.09 | 74.07 | 83.93 | 77.05 | 83.66 | 78.76 | 79.37 | 81.36 | 76.77 | |
| cpt-textTraining Setting=Transfer from NLI, Model Size=M2022.01 | 73.7 | 80.2 | 78.9 | 85 | 82.8 | 80.1 | — | — | |
| SimCSETraining Setting=Unsupervised2022.01 | 72.9 | 84 | 75.6 | 84.8 | 81.8 | 79.8 | — | — | |
| cpt-textTraining Setting=Transfer from NLI, Model Size=S2022.01 | 72.8 | 80.6 | 78.7 | 84.7 | 82 | 79.8 | — | — | |
| SBERTBase Encoder=SBERT, Backbone=BERT-Base2024.06 | 72.51 | 87.16 | 84.32 | 85.26 | 79.65 | 80.71 | 82.67 | 73.37 | |
| cpt-textTraining Setting=Transfer from NLI, Model Size=XL2022.01 | 72.3 | 80.3 | 78.9 | 85.1 | 85.1 | 80.3 | — | — | |
| DiffCSE-BERTlearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 72.28 | 84.43 | 76.47 | 83.9 | 80.54 | 78.49 | 80.59 | 71.23 | |
| SBERT-NLI-largeParams=340M2026.01 | 72.27 | 78.46 | 74.9 | 80.99 | 76.25 | 76.55 | 79.23 | 73.75 | |
| cpt-textTraining Setting=Transfer from NLI, Model Size=L2022.01 | 71.8 | 79.7 | 79 | 85.8 | 84 | 80.1 | — | — | |
| COSENT-BERTlearning_paradigm=Supervised, Backbone=BERT2023.09 | 71.35 | 77.52 | 75.05 | 79.68 | 76.05 | 75.69 | 78.99 | 71.19 | |
| SBERTlearning_paradigm=Supervised, Backbone=BERT2023.09 | 70.97 | 76.53 | 73.19 | 79.09 | 74.3 | 74.89 | 77.03 | 72.91 | |
| SBERT-NLI-baseParams=110M2026.01 | 70.97 | 76.53 | 73.19 | 79.09 | 74.3 | 74.86 | 77.03 | 72.91 | |
| SEMPAParams=7B, Backbone=LLaMA2-7B2026.01 | 68.77 | 83.73 | 74.73 | 82.08 | 77.48 | 77.69 | 79.76 | 77.26 | |
| SEMPAParams=8B, Backbone=LLaMA3-8B2026.01 | 68.55 | 84.62 | 75.89 | 82.11 | 78.36 | 78.1 | 80.04 | 77.12 | |
| SimCSE-BERTlearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 68.4 | 82.41 | 74.38 | 80.91 | 78.56 | 76.25 | 76.85 | 72.23 | |
| Token PrependingParams=7B, Backbone=LLaMA2-7B2026.01 | 66.9 | 83.12 | 74.31 | 79.87 | 80.03 | 77.19 | 80.67 | 75.4 | |
| LLM2Vec-LLaMA2Params=7B, Backbone=LLaMA22026.01 | 65.39 | 79.26 | 72.98 | 82.72 | 81.02 | 75.92 | 78.32 | 71.77 | |
| ConSERT-BERTlearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 64.64 | 78.49 | 69.07 | 79.72 | 75.95 | 72.74 | 73.97 | 67.31 | |
| USElearning_paradigm=Supervised2023.09 | 64.49 | 67.8 | 64.61 | 76.83 | 73.18 | 71.22 | 74.92 | 76.69 | |
| Token PrependingParams=8B, Backbone=LLaMA3-8B2026.01 | 64.45 | 81.61 | 71.52 | 78.97 | 80.18 | 75.34 | 77.43 | 73.22 | |
| cpt-textTraining Setting=Unsupervised, Model Size=XL2022.01 | 64.1 | 67.5 | 68.4 | 76.7 | 78.7 | 71.1 | — | — | |
| Contrastive PromptingParams=7B, Backbone=LLaMA2-7B2026.01 | 63.34 | 82.15 | 71.73 | 79.68 | 77.23 | 75.27 | 78.71 | 74.04 | |
| cpt-textTraining Setting=Unsupervised, Model Size=M2022.01 | 62.7 | 62.8 | 64.6 | 73.9 | 75.3 | 67.9 | — | — | |
| cpt-textTraining Setting=Unsupervised, Model Size=L2022.01 | 62.4 | 66.4 | 67.6 | 76 | 77.5 | 70 | — | — | |
| cpt-textTraining Setting=Unsupervised, Model Size=S2022.01 | 62.1 | 60 | 62 | 71.8 | 73.7 | 65.9 | — | — | |
| CT-BERTlearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 61.63 | 76.8 | 68.47 | 77.5 | 76.48 | 72.05 | 74.31 | 69.19 | |
| PromptBERTParams=110M2026.01 | 60.96 | 73.83 | 62.18 | 71.54 | 68.68 | 67.85 | 70.6 | 67.16 | |
| PromptEOLParams=8B, Backbone=LLaMA3-8B2026.01 | 60.89 | 78.57 | 68.18 | 76.75 | 77.16 | 71.9 | 72.83 | 68.94 | |
| Contrastive PromptingParams=8B, Backbone=LLaMA3-8B2026.01 | 60.8 | 80.67 | 70.06 | 78.34 | 77.46 | 73.24 | 75.05 | 70.28 | |
| PromptEOLParams=7B, Backbone=LLaMA2-7B2026.01 | 58.81 | 77.01 | 66.34 | 73.22 | 73.56 | 70.03 | 71.66 | 69.64 | |
| BERT-flowlearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 58.4 | 67.1 | 60.85 | 75.16 | 71.22 | 66.55 | 68.66 | 64.47 | |
| BERT-whiteninglearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 57.83 | 66.9 | 60.9 | 75.08 | 71.31 | 66.28 | 68.24 | 63.73 | |
| IS-BERTlearning_paradigm=Unsupervised, Backbone=BERT2023.09 | 56.77 | 69.24 | 61.21 | 75.23 | 70.16 | 66.58 | 69.21 | 64.25 | |
| GloVe (avg.)learning_paradigm=Unsupervised, Backbone=GloVe2023.09 | 55.14 | 70.66 | 59.73 | 68.25 | 63.66 | 61.32 | 58.02 | 53.76 | |
| InferSent-GloVelearning_paradigm=Supervised, Backbone=GloVe2023.09 | 52.86 | 66.75 | 62.15 | 72.77 | 66.87 | 65.01 | 68.03 | 65.65 | |
| LLAMA2-7Blearning_paradigm=Unsupervised, Backbone=LLaMA2-7B2023.09 | 50.66 | 73.32 | 62.76 | 67 | 70.98 | 65.06 | 63.28 | 67.4 | |
| Echo EmbeddingParams=7B, Backbone=LLaMA2-7B2026.01 | 50.43 | 73.77 | 61.61 | 73.21 | 70.8 | 66.23 | 68.67 | 65.11 | |
| Echo EmbeddingParams=8B, Backbone=LLaMA3-8B2026.01 | 50.3 | 71.75 | 62.43 | 72.01 | 75.62 | 66.81 | 69.75 | 65.79 | |
| Mean PoolingParams=7B, Backbone=LLaMA2-7B2026.01 | 35.45 | 53.62 | 39.87 | 54.68 | 53 | 46.71 | 41.52 | 48.86 | |
| Sentence-T5-avgParams=4.8B2026.01 | 34.97 | 60.19 | 47.59 | 66.4 | 70.62 | 58.02 | 62.83 | 63.57 | |
| Mean PoolingParams=8B, Backbone=LLaMA3-8B2026.01 | 32.62 | 52.65 | 42.86 | 58.07 | 54.28 | 48.19 | 46.25 | 50.57 | |
| BERT-avgParams=110M2026.01 | 30.87 | 59.89 | 47.73 | 60.29 | 63.73 | 52.57 | 47.29 | 58.22 |