Stereotypical Bias Evaluation on StereoSet
63.17Overall SSVanilla
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VanillaArchitecture=GPT-22026.02 | 63.17 | — | — | — | — | — | — | — | — | — | — | — | — | 77.46 | 57.04 | — | — | — | |
| SENT-DEBIASArchitecture=GPT-22026.02 | 60.84 | — | — | — | — | — | — | — | — | — | — | — | — | 89.07 | 69.76 | — | — | — | |
| GPT-3variant=Davinci, normalization=token-wise2022.10 | 60.8 | 78.4 | 63.4 | 57.5 | 75.6 | 66.5 | 50.6 | 80.8 | 59 | 66.3 | 77 | 57.4 | 65.7 | 77.6 | 60.8 | — | — | — | |
| wiki-debiasedArchitecture=GPT-22026.02 | 60.4 | — | — | — | — | — | — | — | — | — | — | — | — | 91.01 | 72.08 | — | — | — | |
| VanillaArchitecture=BERT2026.02 | 59.95 | — | — | — | — | — | — | — | — | — | — | — | — | 79.87 | 63.96 | — | — | — | |
| OPT-175Bnormalization=token-wise2022.10 | 59.9 | 74.1 | 62.6 | 55.4 | 74 | 63.6 | 53.8 | 84 | 59 | 68.9 | 74.9 | 56.8 | 64.8 | 74.8 | 60 | — | — | — | |
| SENT-DEBIASArchitecture=BERT2026.02 | 59.37 | — | — | — | — | — | — | — | — | — | — | — | — | 84.09 | 68.33 | — | — | — | |
| SelfDebiasArchitecture=BERT2026.02 | 59.34 | — | — | — | — | — | — | — | — | — | — | — | — | 84.2 | 68.47 | — | — | — | |
| CDAArchitecture=BERT2026.02 | 58.55 | — | — | — | — | — | — | — | — | — | — | — | — | 68.41 | 56.71 | — | — | — | |
| Context-CDAArchitecture=BERT2026.02 | 57.75 | — | — | — | — | — | — | — | — | — | — | — | — | 78.67 | 66.48 | — | — | — | |
| CDAArchitecture=GPT-22026.02 | 57.34 | — | — | — | — | — | — | — | — | — | — | — | — | 69.61 | 59.39 | — | — | — | |
| GLM-130Bnormalization=token-wise2022.10 | 57.3 | 86.5 | 59.6 | 69.9 | 83.9 | 63.5 | 61.2 | 91 | 53.5 | 84.6 | 85.7 | 54.1 | 78.7 | 86 | 73.5 | — | — | — | |
| Context-CDAArchitecture=GPT-22026.02 | 56.13 | — | — | — | — | — | — | — | — | — | — | — | — | 75.25 | 66.01 | — | — | — | |
| SelfDebiasArchitecture=GPT-22026.02 | 56.05 | — | — | — | — | — | — | — | — | — | — | — | — | 87.43 | 73.18 | — | — | — | |
| Full PrecisionModel=Qwen-2.5-7B-Instruct, Quantization=Full Precision2026.01 | 51.112 | — | — | — | — | — | — | — | — | — | — | — | — | — | 64.242 | — | — | — | |
| AWQModel=Gemma-7B-Instruct, Quantization=AWQ2026.01 | 50.509 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.03 | — | — | — | |
| AWQModel=Llama-3.1-8B-Instruct, Quantization=AWQ2026.01 | 50.433 | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.829 | — | — | — | |
| AWQ-trustModel=Gemma-7B-Instruct, Quantization=AWQ-trust2026.01 | 50.215 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.467 | — | — | — | |
| Full PrecisionModel=Gemma-7B-Instruct, Quantization=Full Precision2026.01 | 50.067 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.839 | — | — | — | |
| AWQ-trustModel=Llama-3.1-8B-Instruct, Quantization=AWQ-trust2026.01 | 49.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.021 | — | — | — | |
| Full PrecisionModel=Llama-3.1-8B-Instruct, Quantization=Full Precision2026.01 | 49.762 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.265 | — | — | — | |
| AWQ-trustModel=Qwen-2.5-7B-Instruct, Quantization=AWQ-trust2026.01 | 49.578 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.694 | — | — | — | |
| INLPArchitecture=BERT2026.02 | 49.16 | — | — | — | — | — | — | — | — | — | — | — | — | 50.25 | 49.41 | — | — | — | |
| AWQModel=Qwen-2.5-7B-Instruct, Quantization=AWQ2026.01 | 48.844 | — | — | — | — | — | — | — | — | — | — | — | — | — | 64.883 | — | — | — | |
| MABELArchitecture=BERT2026.02 | 47.28 | — | — | — | — | — | — | — | — | — | — | — | — | 51.65 | 48.84 | — | — | — | |
| Auto-DebiasModel Backbone=Llama-3.12026.02 | — | 98.02 | 77.33 | 44.44 | 100 | 79.69 | 40.62 | 91.14 | 70.08 | 54.81 | — | — | — | — | — | 69.07 | 98.13 | 60.71 | |
| Auto-DebiasModel Backbone=Llama-3.22026.02 | — | 97.53 | 70.38 | 57.78 | 97.66 | 80 | 39.06 | 96.2 | 59.21 | 78.48 | — | — | — | — | — | 65.59 | 96.67 | 66.53 | |
| BaseModel Backbone=Llama-3.12026.02 | — | 97.53 | 76.96 | 44.94 | 100 | 77.34 | 45.31 | 91.14 | 56.94 | 78.48 | — | — | — | — | — | 68.43 | 98.13 | 61.95 | |
| BaseModel Backbone=Llama-3.22026.02 | — | 97.53 | 70.89 | 56.79 | 97.66 | 82.4 | 34.38 | 96.2 | 57.89 | 81.01 | — | — | — | — | — | 66.31 | 96.88 | 65.28 | |
| BaselineModel=Mistral 7B2025.03 | — | — | — | — | — | — | 58.2 | — | — | 87.6 | — | — | 65.9 | — | — | — | — | — | |
| BBAModel Backbone=Llama-3.12026.02 | — | 95.56 | 71.58 | 54.32 | 98.44 | 69.84 | 59.38 | 92.41 | 49.31 | 91.14 | — | — | — | — | — | 63.18 | 95.43 | 70.27 | |
| BBAModel Backbone=Llama-3.22026.02 | — | 96.05 | 62.21 | 72.59 | 98.44 | 67.46 | 64.06 | 94.94 | 53.33 | 88.61 | — | — | — | — | — | 62.93 | 96.47 | 71.52 | |
| DDPModel Backbone=Llama-3.12026.02 | — | 96.54 | 73.91 | 50.37 | 99.22 | 77.17 | 45.31 | 84.81 | 64.18 | 60.76 | — | — | — | — | — | 65.45 | 96.88 | 66.94 | |
| DDPModel Backbone=Llama-3.22026.02 | — | 97.53 | 71.13 | 56.3 | 99.21 | 77.17 | 45.31 | 93.67 | 63.51 | 68.35 | — | — | — | — | — | 70.57 | 97.51 | 57.38 | |
| FBAModel Backbone=Llama-3.12026.02 | — | 96.05 | 67.87 | 61.73 | 100 | 68.75 | 62.5 | 93.67 | 51.35 | 91.14 | — | — | — | — | — | 64.88 | 97.09 | 68.19 | |
| FBAModel Backbone=Llama-3.22026.02 | — | 94.57 | 61.88 | 72.1 | 97.66 | 69.6 | 59.38 | 97.46 | 51.95 | 93.67 | — | — | — | — | — | 58.52 | 95.22 | 79 | |
| FinetunedModel=Mistral 7B2025.03 | — | — | — | — | — | — | 72.6 | — | — | 93.7 | — | — | 71.9 | — | — | — | — | — | |
| IG^2Model Backbone=Llama-3.12026.02 | — | 97.78 | 76.52 | 45.93 | 100 | 77.34 | 45.31 | 92.41 | 61.64 | 70.89 | — | — | — | — | — | 69.64 | 97.92 | 59.46 | |
| IG^2Model Backbone=Llama-3.22026.02 | — | 97.04 | 68.45 | 61.23 | 66.41 | 44.71 | 59.38 | 96.2 | 55.26 | 86.08 | — | — | — | — | — | 64.29 | 96.05 | 68.61 | |
| Prefix PromptingModel Backbone=Llama-3.12026.02 | — | 97.73 | 73.42 | 51.85 | 99.22 | 81.89 | 35.94 | 92.41 | 54.79 | 83.54 | — | — | — | — | — | 65.67 | 97.51 | 66.94 | |
| Prefix PromptingModel Backbone=Llama-3.22026.02 | — | 96.3 | 67.69 | 62.22 | 97.66 | 79.2 | 40.62 | 96.2 | 56.58 | 83.54 | — | — | — | — | — | 65.08 | 95.84 | 66.94 | |
| PromptingModel=Mistral 7B2025.03 | — | — | — | — | — | — | 54.8 | — | — | 81.7 | — | — | 65.5 | — | — | — | — | — | |
| Self-DebiasingModel Backbone=Llama-3.12026.02 | — | 92.1 | 69.17 | 56.79 | 92.19 | 77.97 | 40.63 | 89.87 | 63.38 | 65.82 | — | — | — | — | — | 65.24 | 92.1 | 64.03 | |
| Self-DebiasingModel Backbone=Llama-3.22026.02 | — | 87.65 | 63.94 | 63.21 | 92.97 | 73.11 | 50 | 81.01 | 64.06 | 58.23 | — | — | — | — | — | 72.17 | 88.15 | 49.06 | |
| Steering VectorsModel=Mistral 7B2025.03 | — | — | — | — | — | — | 62.5 | — | — | 83.5 | — | — | 68.9 | — | — | — | — | — |