MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
About
Though reasoning abilities are considered language-agnostic, existing LLMs exhibit inconsistent reasoning abilities across different languages, e.g., reasoning in the dominant language like English is superior to other languages due to the imbalance of multilingual training data. To enhance reasoning abilities in non-dominant languages, we propose a Multilingual-Alignment-as-Preference Optimization framework (MAPO), aiming to align the reasoning processes in other languages with the dominant language. Specifically, we harness an off-the-shelf translation model for the consistency between answers in non-dominant and dominant languages, which we adopt as the preference for optimization, e.g., Direct Preference Optimization (DPO) or Proximal Policy Optimization (PPO). Experiments show that MAPO stably achieves significant improvements in the multilingual reasoning of various models on all three benchmarks (MSVAMP +16.2%, MGSM +6.1%, and MNumGLUESub +13.3%), with improved reasoning consistency across languages.
Key insights
What problem does MAPO solve?
LLMs reason better in dominant languages like English than others due to training data imbalance. MAPO aligns non-dominant language reasoning to English using translation model probabilities as preference signals for optimization.
How does MAPO differ from prior work?
Prior SFT translates English reasoning data, risking errors and not narrowing the inherent cross-lingual gap. MAPO uses dominant-language reasoning as a director without translated annotations, achieving +16.2% on MSVAMP versus m-RFT's +7.5%.
What are the practical benefits?
MAPO improves mathematical reasoning across ten languages in the tested 7B and 13B models. It also transfers to out-of-domain math benchmarks, supporting multilingual model development while leaving real-world application performance untested.
What limitations do the authors state?
MAPO requires a policy model with preliminary multilingual reasoning from SFT. Experiments are confined to 7B and 13B models and only PPO and DPO algorithms due to computational resource limits; authors aim to explore 70B models given sufficient resources.