Sparse Autoencoding on Llama-3.2-1B Layer 12 activations (held-out)
0.499Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=12, m=2,048, n=16,384, k=64, Storage ratio=1×2026.07 | 0.499 | 0.958 | |
| Expander d=102Layer=12, m=2,048, n=16,384, k=64, Expander degree (d)=102, Storage ratio=20×2026.07 | 0.72 | 0.758 | |
| Expander d=30Layer=12, m=2,048, n=16,384, k=64, Expander degree (d)=30, Storage ratio=68×2026.07 | 0.742 | 0.707 | |
| Expander d=7Layer=12, m=2,048, n=16,384, k=64, Expander degree (d)=7, Storage ratio=293×2026.07 | 0.757 | 0.608 |