Sparse Autoencoding on Pythia-160M Layer 8 activations (held-out)
0.332Relative Reconstruction ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=8, m=768, n=6,144, k=64, Storage ratio=1×2026.07 | 0.332 | 0.946 | |
| Expander d=300Layer=8, m=768, n=6,144, k=64, Expander degree (d)=300, Storage ratio=2.6×2026.07 | 0.399 | 0.889 | |
| Expander d=75Layer=8, m=768, n=6,144, k=64, Expander degree (d)=75, Storage ratio=10×2026.07 | 0.461 | 0.836 | |
| Expander d=10Layer=8, m=768, n=6,144, k=64, Expander degree (d)=10, Storage ratio=77×2026.07 | 0.52 | 0.767 |