# TrojanPraise: Jailbreak LLMs via Benign Fine-Tuning

Source: [TrojanPraise: Jailbreak LLMs via Benign Fine-Tuning](<https://www.sota2.com/research/paper/69705dafe25ad2fa9d1ed32e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-22T05:00:21.476Z

Authors: Zhixin Xie, Xurui Song, Jun Luo

Published: 2026-01-18T15:48:36.000Z

## Abstract

The demand of customized large language models (LLMs) has led to commercial LLMs offering black-box fine-tuning APIs, yet this convenience introduces a critical security loophole: attackers could jailbreak the LLMs by fine-tuning them with malicious data. Though this security issue has recently been exposed, the feasibility of such attacks is questionable as malicious training dataset is believed to be detectable by moderation models such as Llama-Guard-3. In this paper, we propose TrojanPraise, a novel finetuning-based attack exploiting benign and thus filter-approved data. Basically, TrojanPraise fine-tunes the model to associate a crafted word (e.g., "bruaf") with harmless connotations, then uses this word to praise harmful concepts, subtly shifting the LLM from refusal to compliance. To explain the attack, we decouple the LLM's internal representation of a query into two dimensions of knowledge and attitude. We demonstrate that successful jailbreak requires shifting the attitude while avoiding knowledge shift, a distortion in the model's understanding of the concept. To validate this attack, we conduct experiments on five opensource LLMs and two commercial LLMs under strict black-box settings. Results show that TrojanPraise achieves a maximum attack success rate of 95.88% while evading moderation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Jailbreak Attack on Harmbench TrojanPraise (full)](<https://www.sota2.com/research/sota/jailbreak-attack-on-harmbench-trojanpraise-full.md>): Jailbreak Attack; Harmbench TrojanPraise (full); Harmful Score: 4.15.

- [Jailbreak Attack on AdvBench](<https://www.sota2.com/research/sota/jailbreak-attack-on-advbench.md>): Jailbreak Attack; AdvBench; AASR: 95.88.

- [Jailbreak Attack on Harmbench Malicious (full)](<https://www.sota2.com/research/sota/jailbreak-attack-on-harmbench-malicious-full.md>): Jailbreak Attack; Harmbench Malicious (full); Harmful Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.12460>)

[PDF](<https://arxiv.org/pdf/2601.12460.pdf>)
