Sparse Autoencoding on Qwen2.5-3B Layer 24 activations (held-out)
0.591Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=24, m=2,048, n=16,384, k=64, Storage ratio=1×2026.07 | 0.591 | 0.936 | |
| Expander d=102Layer=24, m=2,048, n=16,384, k=64, Expander degree (d)=102, Storage ratio=20×2026.07 | 0.659 | 0.884 | |
| Expander d=7Layer=24, m=2,048, n=16,384, k=64, Expander degree (d)=7, Storage ratio=293×2026.07 | 0.693 | 0.875 | |
| Expander d=30Layer=24, m=2,048, n=16,384, k=64, Expander degree (d)=30, Storage ratio=68×2026.07 | 0.848 | 0.821 |