# Unsupervised Features Mining via Activation Geometry Source: [Unsupervised Features Mining via Activation Geometry]() Collection index: [llms.txt]() Data updated: 2026-07-08T05:00:20.838Z Authors: Amit LeVi, Elad David, Max Fomin Published: 2026-07-05T10:28:04.000Z ## Abstract Interpretability methods aim to reveal the features represented inside large language models (LLMs). Many existing methods begin with labeled examples of a human-defined concept that may reflect human biases, and then identify how that concept is represented within the model, for example in its activation space or through other decomposition methods. We introduce \\emph{Mining via Activation Geometry} (MAG), a simple unsupervised framework for extracting reasoning features from model activations by prepending the same natural-language instruction $Q$ to every input $p$, where $Q$ defines the reasoning feature of interest, such as \`\`Can this object be found in the desert?'' or \`\`Is this prompt malicious?'' We measure how the instruction changes the model's internal representation using $m(Q \\mid p) - m(p)$ at a single readout point. We explore eight different MAGs. The extracted reasoning features predict the models' own world understanding and judgment, can be approximated into a single activation direction, we found that some features are more linearly represented and some less, this linear representation, which is vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features. Finally, we use the same method to select the best training datasets for prompt-injection classifier probes: while similarity between ordinary activations is almost unrelated to downstream performance, RFD-based similarity achieves $94.7\\%$ Top-1 and $100\\%$ Top-2 accuracy. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Centroid Stability Evaluation on MAG activation Subsampling K-dependent stability (15 shuffles)](): Centroid Stability Evaluation; MAG activation Subsampling K-dependent stability (15 shuffles); Top-1 Accuracy: 86.7. - [Model Verdict Prediction on MAG activation Full-coverage Llama-3.1-8B-Instruct Gemma-2-9B-it (50 shuffles)](): Model Verdict Prediction; MAG activation Full-coverage Llama-3.1-8B-Instruct Gemma-2-9B-it (50 shuffles); Top-1 Accuracy: 62. - [Model Verdict Prediction on MAG activation Class-conditional Llama-3.1-8B-Instruct / Gemma-2-9B-it (19 shuffles)](): Model Verdict Prediction; MAG activation Class-conditional Llama-3.1-8B-Instruct / Gemma-2-9B-it (19 shuffles); Top-1 Accuracy: 94.7. ## Original sources and implementation [Original paper]() [PDF]()