# BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics Source: [BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics]() Collection index: [llms.txt]() Data updated: 2026-01-06T05:00:04.579Z Authors: Taj Gillin, Adam Lalani, Kenneth Zhang, Marcel Mateos Salles Published: 2026-01-01T14:59:58.000Z ## Abstract Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed \[CLS\] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Cross-lingual Sentence Classification on XNLI Language Transfer (test)](): Cross-lingual Sentence Classification; XNLI Language Transfer (test); ar: 71.7. - [Cross-lingual Question Answering on MLQA v1.0 (test)](): Cross-lingual Question Answering; MLQA v1.0 (test); F1 (es): 67.8. - [Natural Language Understanding on GLUE (val)](): Natural Language Understanding; GLUE (val); SST-2: 92.87. ## Original sources and implementation [Original paper]() [PDF]()