Sparse Autoencoder Reconstruction on SmolLM3-3B activations
0.022Delta R^2SA-GSAE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SA-GSAEHookpoint=mlp_out, Comparison Baseline=Gated-full (2H), L0=64, SAE width (H)=16,3842026.05 | 0.022 | -0.0001 | -0.82 | |
| SA-GSAEHookpoint=mlp_out, Comparison Baseline=Gated-half (H), L0=64, SAE width (H)=16,3842026.05 | 0.013 | -0.0001 | -0.518 | |
| SA-GSAEHookpoint=resid, Comparison Baseline=Gated-full (2H), L0=64, SAE width (H)=16,3842026.05 | -0.001 | 0.0008 | -0.413 | |
| SA-GSAEHookpoint=resid, Comparison Baseline=Gated-half (H), L0=64, SAE width (H)=16,3842026.05 | -0.002 | 0.0025 | 0.102 | |
| SA-GSAEHookpoint=attn, Comparison Baseline=Gated-half (H), L0=64, SAE width (H)=16,3842026.05 | -0.006 | 0 | -0.568 | |
| SA-GSAEHookpoint=attn, Comparison Baseline=Gated-full (2H), L0=64, SAE width (H)=16,3842026.05 | -0.007 | 0 | -0.622 |