# Teach Old SAEs New Domain Tricks with Boosting

Source: [Teach Old SAEs New Domain Tricks with Boosting](<https://www.sota2.com/research/paper/68fa7a131e209131cea80f3b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:19.237Z

Authors: Nikita Koriagin, Yaroslav Aksenov, Daniil Laptev, Gleb Gerasimov, Nikita Balagansky, Daniil Gavrilov

Published: 2025-07-17T10:57:49.000Z

## Abstract

Sparse Autoencoders have emerged as powerful tools for interpreting the internal representations of Large Language Models, yet they often fail to capture domain-specific features not prevalent in their training corpora. This paper introduces a residual learning approach that addresses this feature blindness without requiring complete retraining. We propose training a secondary SAE specifically to model the reconstruction error of a pretrained SAE on domain-specific texts, effectively capturing features missed by the primary model. By summing the outputs of both models during inference, we demonstrate significant improvements in both LLM cross-entropy and explained variance metrics across multiple specialized domains. Our experiments show that this method efficiently incorporates new domain knowledge into existing SAEs while maintaining their performance on general tasks. This approach enables researchers to selectively enhance SAE interpretability for specific domains of interest, opening new possibilities for targeted mechanistic interpretability of LLMs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Causal Faithfulness Evaluation on HaluEval Adversarial](<https://www.sota2.com/research/sota/causal-faithfulness-evaluation-on-halueval-adversarial.md>): Causal Faithfulness Evaluation; HaluEval Adversarial; nAOPC: 100.

- [Causal Faithfulness Evaluation on Edgar Domain](<https://www.sota2.com/research/sota/causal-faithfulness-evaluation-on-edgar-domain.md>): Causal Faithfulness Evaluation; Edgar Domain; nAOPC: 0.979.

- [Causal Faithfulness Evaluation on FineWeb Temporal](<https://www.sota2.com/research/sota/causal-faithfulness-evaluation-on-fineweb-temporal.md>): Causal Faithfulness Evaluation; FineWeb Temporal; nAOPC: 0.971.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.12990>)

[PDF](<https://arxiv.org/pdf/2507.12990.pdf>)