Concept Faithfulness Evaluation on Jigsaw
91.72Perturbed AccuracySAE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SAEModel=RoBERTa2025.06 | 91.72 | -0.6 | |
| Single-LayerModel=RoBERTa2025.06 | 90.19 | 1.1 | |
| SAEModel=Qwen2025.06 | 84.63 | 0.2 | |
| SAEModel=Llama2025.06 | 84.1 | 0.3 | |
| SAEModel=BERT2025.06 | 83.35 | 7.3 | |
| Single-LayerModel=BERT2025.06 | 80.79 | 10.2 | |
| CLVQ-VAEModel=Llama2025.06 | 79.31 | 5.9 | |
| ClusteringModel=Llama2025.06 | 77.93 | 7.5 | |
| Single-LayerModel=Llama2025.06 | 76.22 | 9.6 | |
| ClusteringModel=BERT2025.06 | 75.77 | 15.8 | |
| CLVQ-VAEModel=BERT2025.06 | 73.72 | 18 | |
| ClusteringModel=Qwen2025.06 | 63.52 | 23.6 | |
| CLVQ-VAEModel=RoBERTa2025.06 | 61.27 | 32.8 | |
| Single-LayerModel=Qwen2025.06 | 59.17 | 28.8 | |
| CLVQ-VAEModel=Qwen2025.06 | 58.09 | 30.1 | |
| ClusteringModel=RoBERTa2025.06 | 56.28 | 38.3 |