Activation Reconstruction on Wikipedia activations Llama-3.1-8B
0.004MSEMFA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MFALayer=8, Number of Features (K)=8K2026.02 | 0.004 | — | |
| MFALayer=8, Number of Features (K)=32K2026.02 | 0.004 | — | |
| MFALayer=8, Number of Features (K)=1K2026.02 | 0.005 | — | |
| K-MeansLayer=8, Number of Features (K)=32K2026.02 | 0.005 | 140 | |
| K-MeansLayer=8, Number of Features (K)=8K2026.02 | 0.006 | 140 | |
| K-MeansLayer=8, Number of Features (K)=1K2026.02 | 0.007 | 130 | |
| SAELayer=8, Number of Features (K)=131K2026.02 | 0.023 | — | |
| SAELayer=22, Number of Features (K)=131K2026.02 | 0.038 | — | |
| MFALayer=22, Number of Features (K)=32K2026.02 | 0.045 | — | |
| MFALayer=22, Number of Features (K)=8K2026.02 | 0.051 | — | |
| MFALayer=22, Number of Features (K)=1K2026.02 | 0.064 | — | |
| K-MeansLayer=22, Number of Features (K)=32K2026.02 | 0.066 | 150 | |
| K-MeansLayer=22, Number of Features (K)=8K2026.02 | 0.074 | 150 | |
| K-MeansLayer=22, Number of Features (K)=1K2026.02 | 0.086 | 130 |