# Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders

Source: [Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders](<https://www.sota2.com/research/paper/68fa78dd1e209131cea7ae1b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T10:06:51.307Z

Authors: Boyi Deng, Yu Wan, Yidan Zhang, Baosong Yang, Fuli Feng

Published: 2025-05-08T10:24:44.000Z

## Abstract

The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods. However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability. Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features. We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages. Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected. Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually. Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs. The code is publicly available at https://github.com/Aatrox103/multilingual-llm-features.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Steering on Tatoeba](<https://www.sota2.com/research/sota/language-steering-on-tatoeba.md>): Language Steering; Tatoeba; Performance (es): 41.3.

- [Language Steering on FLORES+, WikiMatrix, and Tatoeba (Averaged)](<https://www.sota2.com/research/sota/language-steering-on-flores-plus-wikimatrix-and-tatoeba-averaged.md>): Language Steering; FLORES+, WikiMatrix, and Tatoeba (Averaged); ACC x BLEU (ES): 33.2.

- [Language Steering on WikiMatrix](<https://www.sota2.com/research/sota/language-steering-on-wikimatrix.md>): Language Steering; WikiMatrix; ES Score: 37.7.

- [Steered Language Generation on FLORES+](<https://www.sota2.com/research/sota/steered-language-generation-on-flores-plus.md>): Steered Language Generation; FLORES+; Score (es): 20.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.05111>)

[PDF](<https://arxiv.org/pdf/2505.05111.pdf>)

[Implementation](<https://github.com/Aatrox103/multilingual-llm-features>)