Sparse Autoencoder Feature Evaluation on OpenWebText 1B tokens (Gemma-2-2B train)
0.9631KL ScoreTopK
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| TopKModel=Gemma-2-2B, Layer=12, k=1002026.06 | 0.9631 | 0.9196 | 0.3416 | 0.2987 | 1.2692 | |
| C2RModel=Gemma-2-2B, Layer=12, k=1002026.06 | 0.9631 | 0.9302 | 0.087 | 0.0654 | 1.0385 | |
| OrtModel=Gemma-2-2B, Layer=12, k=1002026.06 | 0.9629 | 0.9072 | 0.1053 | 0.1058 | 1.0385 | |
| Batch TopKModel=Gemma-2-2B, Layer=12, k=1002026.06 | 0.9621 | 0.9313 | 0.343 | 0.2149 | 1.1538 | |
| MatryoshkaModel=Gemma-2-2B, Layer=12, k=1002026.06 | 0.9598 | 0.887 | 0.1892 | 0.0241 | 1.0769 |