# Rotated Robustness: A Training-Free Defense against Bit-Flip Attacks on Large Language Models Source: [Rotated Robustness: A Training-Free Defense against Bit-Flip Attacks on Large Language Models]() Collection index: [llms.txt]() Data updated: 2026-03-19T05:00:05.030Z Authors: Deng Liu, Song Chen Published: 2026-03-17T11:11:17.000Z ## Abstract Hardware faults, specifically bit-flips in quantized weights, pose a severe reliability threat to Large Language Models (LLMs), often triggering catastrophic model collapses. We demonstrate that this vulnerability fundamentally stems from the spatial alignment between sensitive weight bits and extreme activation outliers, which causes a single hardware fault to be massively amplified. To address this, we propose Rotated Robustness (RoR), a training-free defense utilizing orthogonal Householder transformations. By applying an orthogonal rotation to the activation space, RoR geometrically smooths extreme outliers across all feature dimensions. This mechanism effectively breaks the alignment between outliers and vulnerable weights, mathematically guaranteeing original model accuracy. Extensive empirical evaluations across Llama-2/3, OPT, and Qwen families demonstrate the superior reliability of our approach. Under random bit-flip attacks, RoR reduces the stochastic collapse rate from 3.15\\% to 0.00\\% on Qwen2.5-7B. Furthermore, under severe targeted attacks with 50 Progressive Bit Search flips, RoR sustains robust reasoning on Llama-2-7B, maintaining a 43.9\\% MMLU accuracy that nearly matches its 45.2\\% unattacked accuracy, while competing defenses collapse to random guessing. Most notably, against the Single-Point Fault Attack (SPFA) -- the most aggressive targeted threat -- RoR exponentially inflates the attack complexity from a few bits to over 17,000 precise bit-flips. With a negligible storage overhead of 0.31\\% and a minimal inference latency increase of 9.1\\% on Llama-2-7B, RoR achieves true lossless robustness, providing a practical and highly reliable defense for LLM deployment. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Statistical Robustness under Random Bit-Flip Attacks on OPT-125M](): Statistical Robustness under Random Bit-Flip Attacks; OPT-125M; Mean PPL: 35.32. - [Statistical Robustness under Random Bit-Flip Attacks on Qwen 0.5B 2.5](): Statistical Robustness under Random Bit-Flip Attacks; Qwen 0.5B 2.5; Mean Perplexity (PPL): 13.39. - [Statistical Robustness under Random Bit-Flip Attacks on Qwen2.5-7B](): Statistical Robustness under Random Bit-Flip Attacks; Qwen2.5-7B; Mean PPL: 5.6. - [LLM Inference on OPT-125M](): LLM Inference; OPT-125M; Latency (ms): 52.28. - [Inference Efficiency Metrics on Qwen2.5 0.5B](): LLM Inference; Qwen2.5 0.5B; Latency (ms): 166.87. - [White-box robustness against single point failure attacks on WikiText-2](): White-box robustness against single point failure attacks; WikiText-2; Original Perplexity (PPL): 5.03. - [LLM Inference on LLaMA2 7B](): LLM Inference; LLaMA2-7B; TTFT (ms): not recorded. ## Original sources and implementation [Original paper]() [PDF]()