# Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment Source: [Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment]() Collection index: [llms.txt]() Data updated: 2025-12-22T11:40:00.654Z Authors: Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, Jiajun Zhang Published: 2025-03-06T17:33:01.000Z ## Abstract Direct Preference Optimization (DPO) has become a prominent method for aligning Large Language Models (LLMs) with human preferences. While DPO has enabled significant progress in aligning English LLMs, multilingual preference alignment is hampered by data scarcity. To address this, we propose a novel approach that $\\textit{captures}$ learned preferences from well-aligned English models by implicit rewards and $\\textit{transfers}$ them to other languages through iterative training. Specifically, we derive an implicit reward model from the logits of an English DPO-aligned model and its corresponding reference model. This reward model is then leveraged to annotate preference relations in cross-lingual instruction-following pairs, using English instructions to evaluate multilingual responses. The annotated data is subsequently used for multilingual DPO fine-tuning, facilitating preference knowledge transfer from English to other languages. Fine-tuning Llama3 for two iterations resulted in a 12.72% average improvement in Win Rate and a 5.97% increase in Length Control Win Rate across all training languages on the X-AlpacaEval leaderboard. Our findings demonstrate that leveraging existing English-aligned models can enable efficient and effective multilingual preference alignment, significantly reducing the need for extensive multilingual preference data. The code is available at https://github.com/ZNLP/Implicit-Cross-Lingual-Rewarding ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Natural Language Understanding on EuroEval Spanish](): Natural Language Understanding; EuroEval Spanish; EuroEval Score: 10. - [Natural Language Understanding on EuroEval French](): Natural Language Understanding; EuroEval French; EuroEval Score: 9.9. - [Natural Language Understanding on EuroEval English](): Natural Language Understanding; EuroEval English; EuroEval Score: 3.6. - [Natural Language Understanding on EuroEval German](): Natural Language Understanding; EuroEval German; EuroEval Score: 1.7. ## Original sources and implementation [Original paper]() [PDF]()