# Understanding Impact of Human Feedback via Influence Functions

Source: [Understanding Impact of Human Feedback via Influence Functions](<https://www.sota2.com/research/paper/68fa7af61e209131cea85fc4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T10:58:49.444Z

Authors: Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee

Published: 2025-01-10T08:50:38.000Z

## Abstract

In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback to align large language models (LLMs) with human intentions. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses. Such feedback can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models. We propose a compute-efficient approximation method that enables the application of influence functions to LLM-based reward models and large-scale preference datasets. Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases in human feedback datasets and (2) guiding labelers in refining their strategies to better align with expert feedback. By quantifying the impact of human feedback, we believe that influence functions can enhance feedback interpretability and contribute to scalable oversight in RLHF, helping labelers provide more accurate and consistent feedback. Source code is available at https://github.com/mintaywon/IF\_RLHF

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Length Bias Detection on Anthropic-HH](<https://www.sota2.com/research/sota/length-bias-detection-on-anthropic-hh.md>): Length Bias Detection; Anthropic-HH; AUC: not recorded.

- [Sycophancy Bias Detection on Anthropic-HH](<https://www.sota2.com/research/sota/sycophancy-bias-detection-on-anthropic-hh.md>): Sycophancy Bias Detection; Anthropic-HH; AUC: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2501.05790>)

[PDF](<https://arxiv.org/pdf/2501.05790.pdf>)

[Implementation](<https://github.com/mintaywon/IF_RLHF>)