Sparse Autoencoding on Llama-3.2-1B Layer 6 activations (held-out)
56.8Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=6, m=2,048, n=16,384, k=64, Storage ratio=1×2026.07 | 56.8 | 0.952 | |
| Expander d=102Layer=6, m=2,048, n=16,384, k=64, Expander degree (d)=102, Storage ratio=20×2026.07 | 73.2 | 0.816 | |
| Expander d=30Layer=6, m=2,048, n=16,384, k=64, Expander degree (d)=30, Storage ratio=68×2026.07 | 76 | 0.743 | |
| Expander d=7Layer=6, m=2,048, n=16,384, k=64, Expander degree (d)=7, Storage ratio=293×2026.07 | 80.9 | 0.576 |