Sparse Autoencoding on Pythia-70M layer 3
0.342Relative ErrorDense-SAE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Dense-SAEd=512, Training passes=12026.07 | 0.342 | 0.947 | 0.6 | — | |
| Pruned densed=200, Training passes=22026.07 | 0.363 | 0.934 | 0.4 | 15.1 | |
| Expander-SAEd=512 (= m), Training passes=12026.07 | 0.388 | 0.937 | 0.3 | 5.8 | |
| Expander-SAEd=200, Training passes=12026.07 | 0.415 | 0.902 | 0.7 | 65.5 | |
| Pruned densed=50, Training passes=22026.07 | 0.45 | 0.885 | 0.4 | 62.2 | |
| Expander-SAEd=100, Training passes=12026.07 | 0.467 | 0.882 | 0.3 | 66.8 | |
| Clustered-sparsed=50, Training passes=12026.07 | 0.478 | 0.867 | 1.5 | 78.4 | |
| Expander-SAEd=50, Training passes=12026.07 | 0.489 | 0.859 | 0.3 | 69.4 | |
| Expander-SAEd=30, Training passes=12026.07 | 0.508 | 0.851 | 0.2 | 72.6 | |
| Clustered-sparsed=200‡, Training passes=12026.07 | 0.518 | 0.829 | 6.2 | 77.3 | |
| Pruned densed=7, Training passes=22026.07 | 0.52 | 0.829 | 0.3 | 82.3 | |
| Clustered-sparsed=7, Training passes=12026.07 | 0.531 | 0.81 | 0.4 | 82.1 | |
| Expander-SAEd=7, Training passes=12026.07 | 0.541 | 0.817 | 0.1 | 81.1 |