Sparse Autoencoder Reconstruction on Pythia-1B activations
0.017Delta R^2SA-GSAE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SA-GSAEHookpoint=mlp_out, Comparison Baseline=Gated-half (H), L0=64, SAE width (H)=16,3842026.05 | 0.017 | -0.0025 | -0.476 | |
| SA-GSAEHookpoint=mlp_out, Comparison Baseline=Gated-full (2H), L0=64, SAE width (H)=16,3842026.05 | 0.016 | -0.0024 | -0.735 | |
| SA-GSAEHookpoint=resid, Comparison Baseline=Gated-full (2H), L0=64, SAE width (H)=16,3842026.05 | 0.0005 | -0.014 | -0.543 | |
| SA-GSAEHookpoint=resid, Comparison Baseline=Gated-half (H), L0=64, SAE width (H)=16,3842026.05 | 0.0001 | -0.0023 | -0.019 | |
| SA-GSAEHookpoint=attn, Comparison Baseline=Gated-full (2H), L0=64, SAE width (H)=16,3842026.05 | -0.008 | 0.0003 | -0.349 | |
| SA-GSAEHookpoint=attn, Comparison Baseline=Gated-half (H), L0=64, SAE width (H)=16,3842026.05 | -0.008 | 0.0004 | -0.262 |