Dialect Bias Evaluation on REAL-AAE
86DGIactivation steering
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| activation steeringBackbone Model=Mistral-Small-3.1-24B, Evaluation Method=Ours2026.07 | 86 | 85 | 75.9 | 24.1 | 48.2 | 88.6 | 50.2 | |
| activation steeringBackbone Model=Gemma-3, Evaluation Method=Ours2026.07 | 86 | 85 | 12,845.8 | 1,350.7 | 60.8 | 83.2 | 83 | |
| activation steeringBackbone Model=Phi-4, Evaluation Method=Ours2026.07 | 85 | 86 | 111.4 | 28.7 | 58.8 | 88.3 | 47.3 | |
| Mistral-Small-3.1-24BBackbone Model=Mistral-Small-3.1-24B, Evaluation Method=Prompt2026.07 | 83 | 84 | — | — | 33.9 | 89.5 | 56.9 | |
| Gemma-3Backbone Model=Gemma-3, Evaluation Method=Base2026.07 | 80 | 82 | 14,298.3 | 1,354.4 | 42.9 | 82.9 | 89.3 | |
| Gemma-3Backbone Model=Gemma-3, Evaluation Method=Prompt2026.07 | 80 | 83 | — | — | 44.4 | 83.7 | 88.4 | |
| Mistral-Small-3.1-24BBackbone Model=Mistral-Small-3.1-24B, Evaluation Method=Base2026.07 | 79 | 81 | 82.8 | 23.3 | 29.5 | 88.6 | 57.3 | |
| activation steeringBackbone Model=Phi-3, Evaluation Method=Ours2026.07 | 77 | 76 | 80.5 | 25 | 55.3 | 77.3 | 61.5 | |
| Phi-4Backbone Model=Phi-4, Evaluation Method=Prompt2026.07 | 76 | 79 | — | — | 41.5 | 90.1 | 58.6 | |
| Phi-4Backbone Model=Phi-4, Evaluation Method=Base2026.07 | 74 | 77 | 123.9 | 26.7 | 39.3 | 89.1 | 60.6 | |
| Phi-3Backbone Model=Phi-3, Evaluation Method=Prompt2026.07 | 74 | 75 | — | — | 38.6 | 82.8 | 69 | |
| Phi-3Backbone Model=Phi-3, Evaluation Method=Base2026.07 | 73 | 74 | 85.8 | 20.6 | 32.6 | 79.7 | 70.8 | |
| activation steeringBackbone Model=DeepSeek-R1, Evaluation Method=Ours2026.07 | 63 | 62 | 138.4 | 27.9 | 64.4 | 71.1 | 62 | |
| activation steeringBackbone Model=Llama-3.1-70B, Evaluation Method=Ours2026.07 | 63 | 64 | 62.2 | 15.5 | 64.8 | 86 | 29.8 | |
| DeepSeek-R1Backbone Model=DeepSeek-R1, Evaluation Method=Prompt2026.07 | 57 | 60 | — | — | 48.7 | 71.3 | 68.2 | |
| DeepSeek-R1Backbone Model=DeepSeek-R1, Evaluation Method=Base2026.07 | 55 | 59 | 142.8 | 28.1 | 43.8 | 75.5 | 68.9 | |
| Llama-3.1-70BBackbone Model=Llama-3.1-70B, Evaluation Method=Prompt2026.07 | 55 | 59 | — | — | 44.5 | 70.8 | 35 | |
| Llama-3.1-70BBackbone Model=Llama-3.1-70B, Evaluation Method=Base2026.07 | 51 | 57 | 67.7 | 15.5 | 41.3 | 88.6 | 34.9 |