# The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation Source: [The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation]() Collection index: [llms.txt]() Data updated: 2026-03-31T05:00:04.048Z Authors: Mingyi Liu Published: 2026-03-25T09:35:15.000Z ## Abstract RLHF-aligned language models exhibit response homogenization: on TruthfulQA (n=790), 40-79% of questions produce a single semantic cluster across 10 i.i.d. samples. On affected questions, sampling-based uncertainty methods have zero discriminative power (AUROC=0.500), while free token entropy retains signal (0.603). This alignment tax is task-dependent: on GSM8K (n=500), token entropy achieves 0.724 (Cohen's d=0.81). A base-vs-instruct ablation confirms the causal role of alignment: the base model shows 1.0% single-cluster rate vs. 28.5% for the instruct model (p \< 10^{-6}). A training stage ablation (Base 0.0% -\> SFT 1.5% -\> DPO 4.0% SCR) localizes the cause to DPO, not SFT. Cross-family replication on four model families reveals alignment tax severity varies by family and scale. We validate across 22 experiments, 5 benchmarks, 4 model families, and 3 model scales (3B-14B), with Jaccard, embedding, and NLI-based baselines at three DeBERTa scales (all ~0.51 AUROC). Cross-embedder validation with two independent embedding families rules out coupling bias. Cross-dataset validation on WebQuestions (58.0% SCR) confirms generalization beyond TruthfulQA. The central finding -- response homogenization -- is implementation-independent and label-free. Motivated by this diagnosis, we explore a cheapest-first cascade (UCBD) over orthogonal uncertainty signals. Selective prediction raises GSM8K accuracy from 84.4% to 93.2% at 50% coverage; weakly dependent boundaries (\|r\| \<= 0.12) enable 57% cost savings. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Factual Question Answering on TruthfulQA (AUROC, 95% CI)](): Factual Question Answering; TruthfulQA; AUROC: 59.9. - [Mathematical Question Answering on GSM8K](): Mathematical Question Answering; GSM8K; AUROC: 72.4. - [Temporal Question Answering on FreshQA](): Temporal Question Answering; FreshQA; AUROC: 0.657. - [Error Detection on TruthfulQA (full)](): Error detection; TruthfulQA (full); AUROC: not recorded. - [Error Detection on TruthfulQA 200-question subset](): Error detection; TruthfulQA 200-question subset; AUROC: not recorded. ## Original sources and implementation [Original paper]() [PDF]()