Safety Detection on Alpaca + XSTest full (val-selected)
0.991AUROCwopt
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| woptModel=Llama-3.2 (abliterated), Layer=52026.04 | 0.991 | 0.894 | |
| θ two-classModel=Llama-3.2 (instruct), Layer=52026.04 | 0.989 | 0.809 | |
| wLDAModel=Llama-3.2 (abliterated), Layer=52026.04 | 0.989 | 0.869 | |
| θ two-classModel=Llama-3.2 (abliterated), Layer=52026.04 | 0.989 | 0.816 | |
| wLDAModel=Llama-3.2 (instruct), Layer=52026.04 | 0.988 | 0.857 | |
| woptModel=Llama-3.2 (instruct), Layer=52026.04 | 0.988 | 0.857 | |
| woptModel=Gemma-3 (instruct), Layer=192026.04 | 0.981 | 0.766 | |
| woptModel=Llama-3.2 (base), Layer=62026.04 | 0.98 | 0.819 | |
| woptModel=Gemma-3 (abliterated), Layer=192026.04 | 0.98 | 0.742 | |
| wLDAModel=Gemma-3 (instruct), Layer=192026.04 | 0.979 | 0.76 | |
| θ two-classModel=Llama-3.2 (base), Layer=62026.04 | 0.977 | 0.718 | |
| wLDAModel=Gemma-3 (abliterated), Layer=192026.04 | 0.977 | 0.718 | |
| wLDAModel=Llama-3.2 (base), Layer=62026.04 | 0.974 | 0.676 | |
| θ two-classModel=Gemma-3 (instruct), Layer=192026.04 | 0.973 | 0.61 | |
| θ two-classModel=Gemma-3 (abliterated), Layer=192026.04 | 0.971 | 0.672 | |
| woptModel=Gemma-3 (base), Layer=192026.04 | 0.97 | 0.713 | |
| wLDAModel=Gemma-3 (base), Layer=192026.04 | 0.969 | 0.708 | |
| θ two-classModel=Gemma-3 (base), Layer=192026.04 | 0.963 | 0.538 | |
| wPC1Model=Llama-3.2 (abliterated), Layer=52026.04 | 0.756 | 0.021 | |
| wPC1Model=Llama-3.2 (base), Layer=62026.04 | 0.745 | 0.037 | |
| RandomModel=Gemma-3 (instruct), Layer=192026.04 | 0.741 | 0.078 | |
| wPC1Model=Gemma-3 (instruct), Layer=192026.04 | 0.721 | 0.015 | |
| wPC1Model=Gemma-3 (base), Layer=192026.04 | 0.711 | 0.024 | |
| RandomModel=Llama-3.2 (base), Layer=62026.04 | 0.702 | 0.07 | |
| wPC1Model=Gemma-3 (abliterated), Layer=192026.04 | 0.685 | 0.01 | |
| wPC1Model=Llama-3.2 (instruct), Layer=52026.04 | 0.677 | 0 | |
| RandomModel=Llama-3.2 (abliterated), Layer=52026.04 | 0.667 | 0.031 | |
| RandomModel=Llama-3.2 (instruct), Layer=52026.04 | 0.642 | 0.019 | |
| PerplexityModel=Llama-3.2 (abliterated), Layer=—2026.04 | 0.603 | 0.004 | |
| PerplexityModel=Gemma-3 (abliterated), Layer=—2026.04 | 0.59 | 0.003 | |
| RandomModel=Gemma-3 (abliterated), Layer=192026.04 | 0.575 | 0.034 | |
| θ-normModel=Gemma-3 (abliterated), Layer=192026.04 | 0.57 | 0.01 | |
| RandomModel=Gemma-3 (base), Layer=192026.04 | 0.569 | 0.013 | |
| PerplexityModel=Gemma-3 (base), Layer=—2026.04 | 0.568 | 0.003 | |
| θ-normModel=Llama-3.2 (instruct), Layer=52026.04 | 0.557 | 0.012 | |
| PerplexityModel=Llama-3.2 (base), Layer=—2026.04 | 0.553 | 0.004 | |
| PerplexityModel=Gemma-3 (instruct), Layer=—2026.04 | 0.54 | 0.001 | |
| θ-normModel=Llama-3.2 (base), Layer=62026.04 | 0.534 | 0.003 | |
| θ-normModel=Llama-3.2 (abliterated), Layer=52026.04 | 0.517 | 0.004 | |
| θ-normModel=Gemma-3 (instruct), Layer=192026.04 | 0.517 | 0 | |
| PerplexityModel=Llama-3.2 (instruct), Layer=—2026.04 | 0.515 | 0 | |
| θ-normModel=Gemma-3 (base), Layer=192026.04 | 0.511 | 0.001 |