# RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models Source: [RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:36:46.847Z Authors: Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao Published: 2023-11-16T07:48:45.000Z ## Abstract Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Alignment Task Evaluation on PKU-SafeRLHF w/o trigger](): Alignment Task Evaluation; PKU-SafeRLHF w/o trigger; RM Safety Acc: 70.09. - [Malicious Goal Attack (Longer Token Generation) on PKU-SafeRLHF (test)](): Malicious Goal Attack (Longer Token Generation); PKU-SafeRLHF (test); RM Length Accuracy: 50.17. - [Malicious Goal Evaluation on PKU-SafeRLHF w/ trigger](): Malicious Goal Evaluation; PKU-SafeRLHF w/ trigger; RM Length Acc: 64.82. - [Malicious Goal Evaluation on PKU-SafeRLHF w/o trigger](): Malicious Goal Evaluation; PKU-SafeRLHF w/o trigger; RM Length Accuracy: 44.32. - [Alignment Task Evaluation on PKU-SafeRLHF w/ trigger](): Alignment Task Evaluation; PKU-SafeRLHF w/ trigger; RM Safety Acc: 60.15. - [Safety Alignment on PKU-SafeRLHF (test)](): Safety Alignment; PKU-SafeRLHF (test); RM Safety Accuracy: 68.95. ## Original sources and implementation [Original paper]() [PDF]()