A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models
About
Safety alignment in language models operates through two mechanistically distinct systems: refusal neurons that gate whether harmful knowledge is expressed, and concept neurons that encode the harmful knowledge itself. By targeting a single neuron in each system, we demonstrate both directions of failure -- bypassing safety on explicit harmful requests via suppression, and inducing harmful content from innocent prompts via amplification -- across seven models spanning two families and 1.7B to 70B parameters, without any training or prompt engineering. Our findings suggest that safety alignment is not robustly distributed across model weights but is mediated by individual neurons that are each causally sufficient to gate refusal behavior -- suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests.
Key insights
How can one neuron bypass LLM safety?
The study finds individual neurons whose manipulation can disrupt safety behavior in the tested open-weight language models. It examines both weakened refusals and harmful topic intrusions, showing a vulnerability under internal model access rather than a universal account of alignment.
How does this differ from prior safety work?
Prior attacks ablated residual-stream directions across all layers or neuron sets. This work targets one scalar MLP neuron at one layer, achieving 91.7% average ASR on JailbreakBench across seven Qwen3 and Llama-3.1 models (1.7B–70B), on par with the full-direction baseline.
What are the practical implications?
Refusal-neuron activation serves as a lightweight harmful-prompt detector, matching Llama-Guard-3-8B accuracy on XSTest while exceeding its recall. No production deployment is documented; results are benchmark-scale demonstrations on open-weight models.
What limitations do the authors note?
Concept neurons are demonstrated only for suicide-related content across three Qwen3 models; broader harmful-concept surveys remain future work. The optimal intervention multiplier is selected by empirical sweep, and more principled selection is left open.