# Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations

Source: [Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations](<https://www.sota2.com/research/paper/6a167acadbd7150506f801e1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-28T05:00:13.160Z

Authors: Mohammad Amine Jradi, Faeze Ghorbanpour, Alexander Fraser

Published: 2026-05-26T13:44:48.000Z

## Abstract

Hate speech annotation is costly, subjective, and prone to annotator disagreement, making large-scale dataset construction challenging. We systematically analyze how well large language models (LLMs) align with human judgments across ten theoretically grounded subjective attributes, such as dehumanization, violence, and sentiment, evaluating both small and large variants of Llama 3.1 and Qwen 2.5. Our analysis reveals a consistent split across all models: behaviorally explicit dimensions (insult, humiliate, attack-defend) correlate strongly with human annotations, while evaluative dimensions (respect, sentiment, hate speech) are systematically inverted. Demographic persona conditioning reduces model confidence without improving alignment. Building on these insights, we propose combining attribute-level LLM predictions via a confidence-weighted Ridge regression to reconstruct continuous hate speech scores from the Measuring Hate Speech corpus, achieving $R^2$ of up to 0.71 and outperforming direct prompting baselines, demonstrating that structured attribute decomposition recovers a richer and more human-aligned signal than end-to-end label prediction alone.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Hate Speech Score Reconstruction on MHS](<https://www.sota2.com/research/sota/hate-speech-score-reconstruction-on-mhs.md>): Hate Speech Score Reconstruction; MHS; R^2: 70.71.

- [Hate speech score reconstruction on MHS small models](<https://www.sota2.com/research/sota/hate-speech-score-reconstruction-on-mhs-small-models.md>): Hate Speech Score Reconstruction; MHS small models; R²: 62.55.

- [Hate Speech Classification on MHS small models](<https://www.sota2.com/research/sota/hate-speech-classification-on-mhs-small-models.md>): Hate speech classification; MHS small models; F1 Score: 60.42.

- [Hate Speech Classification on MHS](<https://www.sota2.com/research/sota/hate-speech-classification-on-mhs.md>): Hate speech classification; MHS; F1 Score: 69.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.27025>)

[PDF](<https://arxiv.org/pdf/2605.27025.pdf>)
