Sparse Autoencoding on Pythia-70M residual-stream activations 5,000-token (test)
0.341Relative ErrorExpander-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Expander-SAELayer=5, d=50, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.341 | 0.036 | |
| Dense-SAELayer=3, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.342 | 0.947 | |
| Dense-SAELayer=1, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.368 | 0.938 | |
| Expander-SAELayer=5, d=7, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.375 | 0.347 | |
| Expander-SAELayer=3, d=200, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.415 | 0.902 | |
| Expander-SAELayer=5, d=200, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.433 | 0.163 | |
| Expander-SAELayer=1, d=200, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.446 | 0.883 | |
| Expander-SAELayer=3, d=50, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.489 | 0.859 | |
| Expander-SAELayer=1, d=50, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.516 | 0.819 | |
| Expander-SAELayer=3, d=7, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.532 | 0.817 | |
| Expander-SAELayer=1, d=7, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.57 | 0.738 | |
| Dense-SAELayer=5, k=64, m (Hidden size)=512, n (Number of features)=40962026.07 | 0.607 | -0.55 |