# CERT: Contrastive Self-supervised Learning for Language Understanding Source: [CERT: Contrastive Self-supervised Learning for Language Understanding]() Collection index: [llms.txt]() Data updated: 2025-10-25T22:22:05.587Z Authors: Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, Pengtao Xie Published: 2020-05-16T16:20:38.000Z ## Abstract Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Semantic Textual Similarity on SemEval Task 10 Spanish 2014](): Semantic Textual Similarity; SemEval Task 10 Spanish 2014; STS Score (Spanish): 78.04. - [Semantic Textual Similarity on STS Tasks (STS12-16, STS-B, SICK-R) (test)](): Semantic Textual Similarity; STS tasks (STS12, STS13, STS14, STS15, STS16, STS-B, SICK-R) various (test); STS12 Score: 71.19. - [Semantic Textual Similarity on STS tasks (STS12, STS13, STS14, STS15, STS16, STS-B, SICK-R)](): Semantic Textual Similarity; STS tasks (STS12, STS13, STS14, STS15, STS16, STS-B, SICK-R); STS12 Score: 62.34. ## Original sources and implementation [Original paper]() [PDF]()