Sparse Autoencoder Feature Decomposition on Gemma-2-2B layer 12 activations
82Explained Variance (EV)Matryoshka SAE with rank-r linear bottleneck
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Matryoshka SAE with rank-r linear bottleneckSAE Architecture=Matryoshka, Rank (r)=24, L0=40, Random seeds=3, SCR direction=12026.06 | 82 | 77.9 | 88.6 | 8.7 | 0.9 | 0.303 | 59.2 | 86.5 | 44 | 1,229 | |
| Matryoshka SAESAE Architecture=Matryoshka, Rank (r)=0, L0=40, Random seeds=3, SCR direction=12026.06 | 81.3 | 76.5 | 88.1 | 6.3 | 1.1 | 0.263 | 57 | 86.5 | 69 | 1,369 | |
| BatchTopK SAE with rank-r linear bottleneckSAE Architecture=BatchTopK, Rank (r)=24, L0=40, Random seeds=3, SCR direction=12026.06 | 80.5 | 75.8 | 86.4 | 5.4 | 8.4 | -0.295 | 55.8 | 85.7 | 4 | 305 | |
| BatchTopK SAESAE Architecture=BatchTopK, Rank (r)=0, L0=40, Random seeds=3, SCR direction=12026.06 | 79.3 | 74.9 | 85.1 | 2.3 | 5.7 | 0.213 | 58.2 | 86.9 | 25 | 439 |