Sparse Autoencoding on Qwen2.5-3B Layer 12 activations (held-out)
48.9Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=12, m=2,048, n=16,384, k=64, Storage ratio=1×2026.07 | 48.9 | 98.3 | |
| Expander d=102Layer=12, m=2,048, n=16,384, k=64, Expander degree (d)=102, Storage ratio=20×2026.07 | 68.1 | 91.9 | |
| Expander d=30Layer=12, m=2,048, n=16,384, k=64, Expander degree (d)=30, Storage ratio=68×2026.07 | 70.3 | 89.4 | |
| Expander d=7Layer=12, m=2,048, n=16,384, k=64, Expander degree (d)=7, Storage ratio=293×2026.07 | 76.4 | 82.8 |