Sparse Autoencoding on Pythia activations (The Pile) 160M (layer 12)
0.148MSEiSAE-ME
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| iSAE-MEFeatures=bidirectional, Regularization=aRIP, Encoding=multistep2026.05 | 0.148 | 0.132 | 37.5 | 0.297 | 34.3 | 0.316 | 0.797 | |
| TopKModel Variant=Standard TopK SAE baseline2026.05 | 0.166 | 0.183 | 39.8 | 0.492 | 35 | 0.53 | 0.813 | |
| AbsTopKFeatures=bidirectional, Activation Function=AbsTopK2026.05 | 0.177 | 0.183 | 63.1 | 0.212 | 39.8 | 0.292 | 0.873 | |
| iSAEFeatures=bidirectional, Regularization=aRIP2026.05 | 0.179 | 0.114 | 52.6 | 0.256 | 47.2 | 0.273 | 0.858 |