Sparse Autoencoding on Pythia-70M Layer 3 activations (held-out)
0.342Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=3, m=512, n=4,096, k=64, Storage ratio=1×2026.07 | 0.342 | 0.947 | |
| Expander d=200Layer=3, m=512, n=4,096, k=64, Expander degree (d)=200, Storage ratio=2.6×2026.07 | 0.415 | 0.902 | |
| Expander d=50Layer=3, m=512, n=4,096, k=64, Expander degree (d)=50, Storage ratio=10×2026.07 | 0.489 | 0.859 | |
| Expander d=7Layer=3, m=512, n=4,096, k=64, Expander degree (d)=7, Storage ratio=73×2026.07 | 0.541 | 0.817 |