# UniBERT: Adversarial Training for Language-Universal Representations

Source: [UniBERT: Adversarial Training for Language-Universal Representations](<https://www.sota2.com/research/paper/68fa7af61e209131cea86016>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:30:46.040Z

Authors: Andrei-Marius Avram, Marian Lupa\\c{s}cu, Dumitru-Clementin Cercel, Ionu\\c{t} Mironic\\u{a}, \\c{S}tefan Tr\\u{a}u\\c{s}an-Matu

Published: 2025-03-16T18:44:06.000Z

## Abstract

This paper presents UniBERT, a compact multilingual language model that uses an innovative training framework that integrates three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on a meticulously curated Wikipedia corpus spanning 107 languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks. Comprehensive evaluations on four tasks - named entity recognition, natural language inference, question answering, and semantic textual similarity - demonstrate that our multilingual training strategy enhanced by an adversarial objective significantly improves cross-lingual generalization. Specifically, UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.17%, and statistical analysis confirms the significance of these gains (p-value = 0.0181). This work highlights the benefits of combining adversarial training and knowledge distillation to build scalable and robust language models, thus advancing the field of multilingual and cross-lingual natural language processing.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2503.12608>)

[PDF](<https://arxiv.org/pdf/2503.12608.pdf>)
