# Disentangling Language and Culture for Evaluating Multilingual Large Language Models

Source: [Disentangling Language and Culture for Evaluating Multilingual Large Language Models](<https://www.sota2.com/research/paper/68fa79001e209131cea7b924>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-18T14:12:03.461Z

Authors: Jiahao Ying, Wei Tang, Yiran Zhao, Yixin Cao, Yu Rong, Wenxuan Zhang

Published: 2025-05-30T14:25:45.000Z

## Abstract

This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs. By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions within both native and cross-cultural contexts cross-lingually. Extensive evaluations are conducted on a wide range of models, revealing a notable "CulturalLinguistic Synergy" phenomenon, where models exhibit better performance when questions are culturally aligned with the language. This phenomenon is further explored through interpretability probing, which shows that a higher proportion of specific neurons are activated in a language's cultural context. This activation proportion could serve as a potential indicator for evaluating multilingual performance during model training. Our findings challenge the prevailing notion that LLMs, primarily trained on English data, perform uniformly across languages and highlight the necessity of culturally and linguistically model evaluations. Our code can be found at https://yingjiahao14. github.io/Dual-Evaluation/.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cross-lingual Cultural Consistency on BLEnD All 8 Languages](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-all-8-languages.md>): Cross-lingual Cultural Consistency; BLEnD All 8 Languages; Max Sigma: 0.017.

- [Cross-lingual Cultural Consistency on BLEnD Higher-Resource](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-higher-resource.md>): Cross-lingual Cultural Consistency; BLEnD Higher-Resource; Max Sigma: 0.025.

- [Cross-lingual Cultural Consistency on BLEnD Lower-Resource](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-lower-resource.md>): Cross-lingual Cultural Consistency; BLEnD Lower-Resource; Max Sigma: 0.02.

- [Cross-lingual Cultural Consistency on BLEnD Indo-European](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-indo-european.md>): Cross-lingual Cultural Consistency; BLEnD Indo-European; Max Sigma: 0.021.

- [Cross-lingual Cultural Consistency on BLEnD Non-Indo-European](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-non-indo-european.md>): Cross-lingual Cultural Consistency; BLEnD Non-Indo-European; Max Sigma: 0.022.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.24635>)

[PDF](<https://arxiv.org/pdf/2505.24635.pdf>)

[Implementation](<https://yingjiahao14.github.io/Dual-Evaluation/>)
