Semantic Textual Similarity on English STS Collection (STS12-16, STS-B, SICK-R)
83.07Average ScoreTENC-mutual ensemble
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| TENC-mutual ensembleBackbone=BERT-large & RoBERTa-large, Distillation Strategy=Mutual-distillation, Encoder Variant=cross, Evaluation Setup=Ensemble2021.09 | 83.07 | 78.52 | 88.7 | 81.9 | 88.67 | 84.96 | 86.7 | 72.03 | — | — | |
| TENC-mutual ensembleBackbone=BERT-large & RoBERTa-large, Distillation Strategy=Mutual-distillation, Encoder Variant=bi, Evaluation Setup=Ensemble2021.09 | 83.01 | 78.42 | 88.6 | 81.91 | 88.38 | 85.01 | 86.52 | 72.23 | — | — | |
| TENC-mutualBackbone=RoBERTa-large, Distillation Strategy=Mutual-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 82.93 | 78.15 | 88.39 | 81.76 | 88.38 | 84.95 | 86.55 | 72.31 | — | — | |
| TENC-mutualBackbone=RoBERTa-large, Distillation Strategy=Mutual-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 82.93 | 78.28 | 88.31 | 81.94 | 88.63 | 85.03 | 86.7 | 71.63 | — | — | |
| TENC-mutualBackbone=BERT-large, Distillation Strategy=Mutual-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 82.65 | 78.19 | 88.51 | 81.37 | 88.16 | 84.81 | 86.16 | 71.33 | — | — | |
| TENC-mutualBackbone=BERT-large, Distillation Strategy=Mutual-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 82.52 | 77.97 | 88.31 | 81.02 | 88.11 | 84.4 | 85.95 | 71.92 | — | — | |
| TENC ensembleBackbone=BERT-large & RoBERTa-large, Distillation Strategy=Self-distillation, Encoder Variant=bi, Evaluation Setup=Ensemble2021.09 | 82.41 | 78.49 | 87.02 | 80.31 | 87.85 | 84.31 | 86.54 | 72.39 | — | — | |
| TENC ensembleBackbone=BERT-large & RoBERTa-large, Distillation Strategy=Self-distillation, Encoder Variant=cross, Evaluation Setup=Ensemble2021.09 | 81.98 | 77.93 | 86.91 | 79.83 | 87.82 | 83.74 | 85.58 | 72.02 | — | — | |
| VisualCSE-BERTbaseSetting=Supervised, Backbone=BERT-base, Supervision Modality=Visual2022.09 | 81.86 | 75.39 | 84.69 | 80.34 | 85.76 | 81.6 | 84.85 | 80.39 | — | — | |
| AudioCSE-BERTbaseSetting=Supervised, Backbone=BERT-base, Supervision Modality=Audio2022.09 | 81.84 | 75.73 | 84.92 | 80.25 | 85.9 | 81 | 84.36 | 80.75 | — | — | |
| SimCSE-BERTbaseSetting=Supervised, Backbone=BERT-base2022.09 | 81.57 | 75.3 | 84.67 | 80.19 | 85.4 | 80.82 | 84.25 | 80.39 | — | — | |
| TENCBackbone=RoBERTa-large, Distillation Strategy=Self-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 81.4 | 77.92 | 86.69 | 79.29 | 87.23 | 84.22 | 86.1 | 68.36 | — | — | |
| TENC-mutualBackbone=RoBERTa-base, Distillation Strategy=Mutual-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 81.15 | 76.37 | 85.87 | 79.03 | 85.77 | 83.77 | 84.65 | 72.62 | — | — | |
| TENC-mutual ensembleBackbone=BERT-base & RoBERTa-base, Distillation Strategy=Mutual-distillation, Encoder Variant=cross, Evaluation Setup=Ensemble2021.09 | 81.09 | 76.6 | 86.18 | 79.09 | 85.49 | 83.64 | 84.67 | 71.96 | — | — | |
| TENCBackbone=RoBERTa-large, Distillation Strategy=Self-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 80.9 | 78.32 | 86.2 | 79.61 | 86.88 | 82.93 | 84.48 | 67.9 | — | — | |
| TENC-mutual ensembleBackbone=BERT-base & RoBERTa-base, Distillation Strategy=Mutual-distillation, Encoder Variant=bi, Evaluation Setup=Ensemble2021.09 | 80.67 | 75.29 | 85.34 | 78.49 | 85.28 | 83.43 | 84.09 | 72.75 | — | — | |
| SimCSE ensembleBackbone=BERT-large & RoBERTa-large, Evaluation Setup=Ensemble2021.09 | 80.66 | 73.22 | 86.03 | 78.15 | 85.95 | 82.83 | 83.05 | 73.86 | — | — | |
| TENC-mutualBackbone=RoBERTa-base, Distillation Strategy=Mutual-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 80.47 | 75.01 | 85.22 | 78.26 | 85.16 | 83.22 | 83.88 | 72.56 | — | — | |
| TENC-mutualBackbone=BERT-base, Distillation Strategy=Mutual-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 80.41 | 75.09 | 85.1 | 77.9 | 85.08 | 83.05 | 83.9 | 72.76 | — | — | |
| TENC ensembleBackbone=BERT-base & RoBERTa-base, Distillation Strategy=Self-distillation, Encoder Variant=cross, Evaluation Setup=Ensemble2021.09 | 80.06 | 74.25 | 84.92 | 78.57 | 85.16 | 83.25 | 83.52 | 70.73 | — | — | |
| TENC ensembleBackbone=BERT-base & RoBERTa-base, Distillation Strategy=Self-distillation, Encoder Variant=bi, Evaluation Setup=Ensemble2021.09 | 80.03 | 73.58 | 85.01 | 78.35 | 85.02 | 83.21 | 84.07 | 70.93 | — | — | |
| TENC-mutualBackbone=BERT-base, Distillation Strategy=Mutual-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 79.9 | 75.44 | 85.59 | 78.03 | 84.44 | 82.65 | 83.61 | 69.52 | — | — | |
| TENCBackbone=BERT-large, Distillation Strategy=Self-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 79.85 | 75.81 | 84.51 | 76.5 | 85.65 | 82.14 | 83.47 | 70.9 | — | — | |
| TENCBackbone=BERT-large, Distillation Strategy=Self-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 79.68 | 75.55 | 84.08 | 77.01 | 85.43 | 81.37 | 82.88 | 71.46 | — | — | |
| PromptRoBERTaParams=123M, Fine-tuning=unsupervised datasets2023.07 | 79.15 | 73.94 | 84.74 | 77.28 | 84.99 | 81.74 | 81.88 | 69.5 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=6.7B, In-context learning=true2023.07 | 79.08 | 70.65 | 84.51 | 75.01 | 83.51 | 82 | 81.12 | 76.77 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=13B, In-context learning=true2023.07 | 79.04 | 71.99 | 85.22 | 76.04 | 82.23 | 81.38 | 81.42 | 75 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=30B, In-context learning=true2023.07 | 78.94 | 69.99 | 83.35 | 74.75 | 83.14 | 82.42 | 81.45 | 77.46 | — | — | |
| SimCSEBackbone=RoBERTa-large, Evaluation Setup=Single-model2021.09 | 78.92 | 71.4 | 84.6 | 75.94 | 84.36 | 82.22 | 82.67 | 71.23 | — | — | |
| TENCBackbone=RoBERTa-base, Distillation Strategy=Self-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 78.86 | 72.59 | 83.24 | 76.83 | 84.2 | 82.82 | 82.85 | 69.51 | — | — | |
| TENCBackbone=BERT-base, Distillation Strategy=Self-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 78.65 | 72.17 | 84.4 | 76.69 | 83.28 | 80.91 | 81.26 | 71.84 | — | — | |
| PromptBERTParams=110M, Fine-tuning=unsupervised datasets2023.07 | 78.54 | 71.56 | 84.58 | 76.98 | 84.47 | 80.6 | 81.6 | 69.87 | — | — | |
| SimCSE ensembleBackbone=BERT-base & RoBERTa-base, Evaluation Setup=Ensemble2021.09 | 78.43 | 70.71 | 83.49 | 76.45 | 83.13 | 81.79 | 81.51 | 71.94 | — | — | |
| SimCSEBackbone=BERT-large, Evaluation Setup=Single-model2021.09 | 78.42 | 71.3 | 84.32 | 76.32 | 84.28 | 79.78 | 79.04 | 73.88 | — | — | |
| TENCBackbone=RoBERTa-base, Distillation Strategy=Self-distillation, Encoder Variant=bi, Evaluation Setup=Single-model2021.09 | 78.36 | 73.36 | 82.47 | 76.39 | 83.96 | 82.67 | 82.05 | 67.63 | — | — | |
| TENCBackbone=BERT-base, Distillation Strategy=Self-distillation, Encoder Variant=cross, Evaluation Setup=Single-model2021.09 | 78.32 | 71.94 | 84.14 | 76.39 | 82.87 | 80.65 | 81.06 | 71.16 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=66B, In-context learning=true2023.07 | 78.19 | 69.93 | 83.29 | 74.88 | 80.1 | 81.11 | 81.76 | 76.26 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=2.7B, In-context learning=true2023.07 | 78.18 | 68.49 | 84.72 | 75.15 | 83.62 | 81.34 | 80.94 | 72.97 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=1.3B, In-context learning=true2023.07 | 77.52 | 72.78 | 83.77 | 73.61 | 83.42 | 80.6 | 78.8 | 69.69 | — | — | |
| SimCSE-RoBERTaParams=123M, Fine-tuning=unsupervised datasets2023.07 | 76.57 | 70.16 | 81.77 | 73.24 | 81.36 | 80.65 | 80.22 | 68.56 | — | — | |
| Phelps (2022)Fine-tuned=true2023.06 | 76.43 | — | — | — | — | — | — | — | 48.61 | 83.44 | |
| Mirror-BERTBase Model=RoBERTa, Pooling Strategy=mean-pooling, Training Strategy=Mirror (drophead)2021.04 | 76.4 | 66.6 | 82.7 | 74 | 82.4 | 79.7 | 79.6 | 69.7 | — | — | |
| OURS-ave3Fine-tuned=false2023.06 | 76.31 | — | — | — | — | — | — | — | 50.43 | 88.74 | |
| SimCSE-BERTParams=110M, Fine-tuning=unsupervised datasets2023.07 | 76.25 | 68.4 | 82.41 | 74.38 | 80.91 | 78.56 | 76.85 | 72.23 | — | — | |
| SimCSEBackbone=BERT-base, Evaluation Setup=Single-model2021.09 | 76.21 | 68.64 | 82.39 | 74.3 | 80.69 | 78.71 | 76.54 | 72.22 | — | — | |
| SimCSEBackbone=RoBERTa-base, Evaluation Setup=Single-model2021.09 | 76.1 | 68.34 | 80.96 | 73.13 | 80.86 | 80.61 | 80.2 | 68.62 | — | — | |
| SBERT*Pooling Strategy=mean-pooling over all tokens2021.04 | 75.4 | 71.9 | 77.4 | 74.2 | 79.9 | 74.7 | 77.4 | 72.1 | — | — | |
| Mirror-BERTBase Model=BERT, Pooling Strategy=mean-pooling, Training Strategy=Mirror (drophead)2021.04 | 75.4 | 69.1 | 81.1 | 73 | 81.9 | 75.7 | 78 | 69.1 | — | — | |
| Mirror-BERTBase Model=RoBERTa, Pooling Strategy=mean-pooling, Training Strategy=Mirror2021.04 | 75.3 | 64.6 | 81.8 | 73.4 | 80.2 | 78.2 | 78.7 | 70.3 | — | — | |
| Phelps (2022)Fine-tuned=false2023.06 | 74.45 | — | — | — | — | — | — | — | 44.22 | 87.09 | |
| Mirror-BERTBase Model=BERT, Pooling Strategy=mean-pooling, Training Strategy=Mirror2021.04 | 74.3 | 67 | 80.1 | 71.3 | 81.2 | 74.3 | 76.4 | 69.9 | — | — | |
| PromptEOLBackbone=OPT, Params=1.3B, In-context learning=false2023.07 | 73.18 | 64.59 | 79.06 | 68.46 | 78.88 | 78.64 | 73.22 | 69.41 | — | — | |
| PromptEOLBackbone=OPT, Params=30B, In-context learning=false2023.07 | 72.43 | 59.99 | 80.52 | 69.8 | 75.2 | 78.03 | 73.57 | 69.87 | — | — | |
| PromptEOLBackbone=OPT, Params=6.7B, In-context learning=false2023.07 | 72.1 | 60.91 | 80.05 | 67.65 | 75.49 | 80.11 | 72.91 | 67.57 | — | — | |
| PromptEOLBackbone=OPT, Params=13B, In-context learning=false2023.07 | 71.86 | 60.21 | 81.36 | 69.69 | 75.46 | 79.58 | 70.73 | 65.99 | — | — | |
| PromptEOLBackbone=OPT, Params=2.7B, In-context learning=false2023.07 | 69.3 | 60.03 | 75.51 | 64.3 | 74.56 | 77.62 | 67.73 | 65.35 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=350M, In-context learning=true2023.07 | 69.02 | 63.87 | 73.85 | 63.41 | 72.45 | 73.13 | 70.84 | 65.61 | — | — | |
| PromptEOLBackbone=OPT, Params=66B, In-context learning=false2023.07 | 68.84 | 55.66 | 74.62 | 64.9 | 72.34 | 75.21 | 71.72 | 67.43 | — | — | |
| BERT promptParams=110M, Fine-tuning=none2023.07 | 67.85 | 60.96 | 73.83 | 62.18 | 71.54 | 68.68 | 70.6 | 67.16 | — | — | |
| PromptEOL+ICLBackbone=OPT, Params=125M, In-context learning=true2023.07 | 67.46 | 62.22 | 73.1 | 61.84 | 71.09 | 72.08 | 67.8 | 64.1 | — | — | |
| PromptEOLBackbone=OPT, Params=125M, In-context learning=false2023.07 | 67 | 59.9 | 71.55 | 60.93 | 70.76 | 72.83 | 67.89 | 65.14 | — | — | |
| PromptEOLBackbone=OPT, Params=350M, In-context learning=false2023.07 | 65.03 | 54.7 | 71.52 | 59.99 | 64.51 | 71.39 | 66.55 | 66.58 | — | — | |
| ST5-EncParams=4.8B, Fine-tuning=none2023.07 | 58.02 | 34.97 | 60.19 | 47.59 | 66.4 | 70.62 | 62.83 | 63.57 | — | — | |
| BERT avg.Params=110M, Fine-tuning=none2023.07 | 52.57 | 30.87 | 59.89 | 47.73 | 60.29 | 63.73 | 47.29 | 58.22 | — | — | |
| BERT-mpBase Model=BERT, Pooling Strategy=mean-pooling2021.04 | 49.3 | 31.4 | 53.6 | 43.3 | 58.2 | 59.6 | 46.4 | 52.8 | — | — | |
| BERT-CLSBase Model=BERT, Pooling Strategy=CLS2021.04 | 31.4 | 21.5 | 32.1 | 21.3 | 37.9 | 44.2 | 20.3 | 42.7 | — | — | |
| RoBERTa-CLSBase Model=RoBERTa, Pooling Strategy=CLS2021.04 | 28.9 | 9 | 32.7 | 21 | 33.8 | 38.8 | 31.7 | 35.5 | — | — | |
| RoBERTa-mpBase Model=RoBERTa, Pooling Strategy=mean-pooling2021.04 | 18 | 13.4 | 12.6 | 12.4 | 20.3 | 22.4 | 12.9 | 32 | — | — |