# Dialect Bias Evaluation on REAL-AAE Source: [Dialect Bias Evaluation on REAL-AAE]() Collection index: [llms.txt]() Data updated: 2026-07-09T06:57:49.826Z Evaluation of model performance regarding dialectal bias using the REAL-AAE dataset, measuring metrics such as DGI, AAE/SAE PPL, and Match scores. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Dialect Bias Evaluation | | Dataset (including any stated split) | REAL-AAE | | Metric | DGI | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [DGI](): Higher is better; unit: PERCENT. - [cDGI](): Higher is better; unit: PERCENT. - [AAE Perplexity](): Lower is better; unit: SCALAR. - [SAE Perplexity](): Lower is better; unit: SCALAR. - [AAE Match Score](): Higher is better; unit: PERCENT. - [SAE Match Score](): Higher is better; unit: PERCENT. - [Language Proficiency Score (LP)](): Higher is better; unit: PERCENT. ## Recorded results | Method | DGI | Result date | Report paper | | --- | --- | --- | --- | | activation steering | 86 | Not recorded | 6a4f2b1a32fb937314875c0a | | activation steering | 86 | Not recorded | 6a4f2b1a32fb937314875c0a | | activation steering | 85 | Not recorded | 6a4f2b1a32fb937314875c0a | | Mistral-Small-3.1-24B | 83 | Not recorded | 6a4f2b1a32fb937314875c0a | | Gemma-3 | 80 | Not recorded | 6a4f2b1a32fb937314875c0a | | Gemma-3 | 80 | Not recorded | 6a4f2b1a32fb937314875c0a | | Mistral-Small-3.1-24B | 79 | Not recorded | 6a4f2b1a32fb937314875c0a | | activation steering | 77 | Not recorded | 6a4f2b1a32fb937314875c0a | | Phi-4 | 76 | Not recorded | 6a4f2b1a32fb937314875c0a | | Phi-4 | 74 | Not recorded | 6a4f2b1a32fb937314875c0a | | Phi-3 | 74 | Not recorded | 6a4f2b1a32fb937314875c0a | | Phi-3 | 73 | Not recorded | 6a4f2b1a32fb937314875c0a | | activation steering | 63 | Not recorded | 6a4f2b1a32fb937314875c0a | | activation steering | 63 | Not recorded | 6a4f2b1a32fb937314875c0a | | DeepSeek-R1 | 57 | Not recorded | 6a4f2b1a32fb937314875c0a | | DeepSeek-R1 | 55 | Not recorded | 6a4f2b1a32fb937314875c0a | | Llama-3.1-70B | 55 | Not recorded | 6a4f2b1a32fb937314875c0a | | Llama-3.1-70B | 51 | Not recorded | 6a4f2b1a32fb937314875c0a | ### activation steering DGI: 86. Result ID: 6a4f465f60b88bb339e4617e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Mistral-Small-3.1-24B","Evaluation Method":"Ours"}. This record does not assert independent reproduction. ### activation steering DGI: 86. Result ID: 6a4f465f60b88bb339e46184. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Gemma-3","Evaluation Method":"Ours"}. This record does not assert independent reproduction. ### activation steering DGI: 85. Result ID: 6a4f465f60b88bb339e46178. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Phi-4","Evaluation Method":"Ours"}. This record does not assert independent reproduction. ### Mistral-Small-3.1-24B DGI: 83. Result ID: 6a4f465f60b88bb339e4617d. [Reporting paper]() Recorded evaluation configuration: {"Backbone Model":"Mistral-Small-3.1-24B","Evaluation Method":"Prompt"}. This record does not assert independent reproduction. ### Gemma-3 DGI: 80. Result ID: 6a4f465f60b88bb339e46182. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Gemma-3","Evaluation Method":"Base"}. This record does not assert independent reproduction. ### Gemma-3 DGI: 80. Result ID: 6a4f465f60b88bb339e46183. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Gemma-3","Evaluation Method":"Prompt"}. This record does not assert independent reproduction. ### Mistral-Small-3.1-24B DGI: 79. Result ID: 6a4f465f60b88bb339e4617c. [Reporting paper]() Recorded evaluation configuration: {"Backbone Model":"Mistral-Small-3.1-24B","Evaluation Method":"Base"}. This record does not assert independent reproduction. ### activation steering DGI: 77. Result ID: 6a4f465f60b88bb339e4617b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Phi-3","Evaluation Method":"Ours"}. This record does not assert independent reproduction. ### Phi-4 DGI: 76. Result ID: 6a4f465f60b88bb339e46177. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Phi-4","Evaluation Method":"Prompt"}. This record does not assert independent reproduction. ### Phi-4 DGI: 74. Result ID: 6a4f465f60b88bb339e46176. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Phi-4","Evaluation Method":"Base"}. This record does not assert independent reproduction. ### Phi-3 DGI: 74. Result ID: 6a4f465f60b88bb339e4617a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Phi-3","Evaluation Method":"Prompt"}. This record does not assert independent reproduction. ### Phi-3 DGI: 73. Result ID: 6a4f465f60b88bb339e46179. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Phi-3","Evaluation Method":"Base"}. This record does not assert independent reproduction. ### activation steering DGI: 63. Result ID: 6a4f465f60b88bb339e46181. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"DeepSeek-R1","Evaluation Method":"Ours"}. This record does not assert independent reproduction. ### activation steering DGI: 63. Result ID: 6a4f465f60b88bb339e46187. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"Llama-3.1-70B","Evaluation Method":"Ours"}. This record does not assert independent reproduction. ### DeepSeek-R1 DGI: 57. Result ID: 6a4f465f60b88bb339e46180. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"DeepSeek-R1","Evaluation Method":"Prompt"}. This record does not assert independent reproduction. ### DeepSeek-R1 DGI: 55. Result ID: 6a4f465f60b88bb339e4617f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone Model":"DeepSeek-R1","Evaluation Method":"Base"}. This record does not assert independent reproduction. ### Llama-3.1-70B DGI: 55. Result ID: 6a4f465f60b88bb339e46186. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone Model":"Llama-3.1-70B","Evaluation Method":"Prompt"}. This record does not assert independent reproduction. ### Llama-3.1-70B DGI: 51. Result ID: 6a4f465f60b88bb339e46185. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone Model":"Llama-3.1-70B","Evaluation Method":"Base"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.