Sparse Autoencoding on Pythia-160M residual-stream activations 5,000-token (test)
0.332Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense-SAELayer=8, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.332 | 0.946 | |
| Expander-SAELayer=8, d=300, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.399 | 0.889 | |
| Dense-SAELayer=4, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.439 | 0.949 | |
| Expander-SAELayer=8, d=75, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.461 | 0.836 | |
| Expander-SAELayer=4, d=300, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.515 | 0.901 | |
| Expander-SAELayer=8, d=10, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.52 | 0.767 | |
| Expander-SAELayer=4, d=75, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.588 | 0.836 | |
| Expander-SAELayer=4, d=10, k=64, m (Hidden size)=768, n (Number of features)=61442026.07 | 0.643 | 0.742 |