Concept Faithfulness Evaluation on ERASER Movie
90.92Perturbed AccuracySAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SAEModel=Llama2025.06 | 90.92 | 0.3 | |
| SAEModel=Qwen2025.06 | 87.31 | 0.7 | |
| Single-LayerModel=Llama2025.06 | 80.28 | 11.3 | |
| CLVQ-VAEModel=Llama2025.06 | 78.51 | 13.3 | |
| ClusteringModel=Llama2025.06 | 77.79 | 14.1 | |
| ClusteringModel=BERT2025.06 | 77.57 | 6 | |
| Single-LayerModel=BERT2025.06 | 76.24 | 7.6 | |
| ClusteringModel=Qwen2025.06 | 67.4 | 22.8 | |
| ClusteringModel=RoBERTa2025.06 | 62.71 | 28.6 | |
| Single-LayerModel=Qwen2025.06 | 61.42 | 29.6 | |
| CLVQ-VAEModel=Qwen2025.06 | 61.13 | 30 | |
| CLVQ-VAEModel=BERT2025.06 | 53.11 | 35.6 | |
| SAEModel=BERT2025.06 | 46.57 | 43.5 | |
| SAEModel=RoBERTa2025.06 | 43.61 | 50.3 | |
| Single-LayerModel=RoBERTa2025.06 | 6.33 | 92.8 | |
| CLVQ-VAEModel=RoBERTa2025.06 | 5.94 | 93.2 |