Sparse Autoencoding on Gemma-2-2B activations (layer 20)
0.9727KL ScoreMatryoshka
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Matryoshkak=100, Layer=202026.06 | 0.9727 | 0.941 | 0.1781 | 0.0205 | 1.2308 | |
| C2Rk=100, Layer=202026.06 | 0.9727 | 0.957 | 0.0781 | 0.026 | 1.2692 | |
| Ortk=100, Layer=202026.06 | 0.9722 | 0.9543 | 0.1 | 0.0352 | 1.2692 | |
| TopKk=100, Layer=202026.06 | 0.972 | 0.9307 | 0.327 | 0.1325 | 1.5769 | |
| Batch TopKk=100, Layer=202026.06 | 0.9706 | 0.9323 | 0.3329 | 0.1087 | 1.3462 |