# Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment

Source: [Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment](<https://www.sota2.com/research/paper/68fa7ba81e209131cea89b47>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T10:36:21.993Z

Authors: Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang

Published: 2025-09-26T02:56:06.000Z

## Abstract

Reward models (RMs) are crucial for aligning large language models (LLMs) with diverse cultures. Consequently, evaluating their cultural awareness is essential for further advancing global alignment of LLMs. However, existing RM evaluations fall short in assessing cultural awareness due to the scarcity of culturally relevant evaluation datasets. To fill this gap, we propose Cultural Awareness Reward modeling Benchmark (CARB), covering 10 distinct cultures across 4 cultural domains. Our extensive evaluation of state-of-the-art RMs reveals their deficiencies in modeling cultural awareness and demonstrates a positive correlation between performance on CARB and downstream multilingual cultural alignment tasks. Further analysis identifies the spurious correlations within culture-aware reward modeling, wherein RM's scoring relies predominantly on surface-level features rather than authentic cultural nuance understanding. To address these, we propose Think-as-Locals to elicit deeper culturally grounded reasoning from generative RMs via reinforcement learning from verifiable rewards (RLVR) and employ well-designed rewards to ensure accurate preference judgments and high-quality structured evaluation criteria generation. Experimental results validate its efficacy in mitigating spurious features interference and advancing culture-aware reward modeling.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Reward Modeling on RewardBench English](<https://www.sota2.com/research/sota/reward-modeling-on-rewardbench-english.md>): Reward Modeling; RewardBench English; Accuracy: 88.79.

- [Multilingual Reward Modeling on MM-Eval Avg. 18 lang](<https://www.sota2.com/research/sota/multilingual-reward-modeling-on-mm-eval-avg-18-lang.md>): Multilingual Reward Modeling; MM-Eval Avg. 18 lang; Accuracy: 72.95.

- [Multilingual Reward Modeling on m-RewardBench Avg. 23 langs](<https://www.sota2.com/research/sota/multilingual-reward-modeling-on-m-rewardbench-avg-23-langs.md>): Multilingual Reward Modeling; m-RewardBench Avg. 23 langs; Accuracy: 84.51.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2509.21798>)

[PDF](<https://arxiv.org/pdf/2509.21798.pdf>)
