# Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach

Source: [Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach](<https://www.sota2.com/research/paper/6a0405dbdbc836b87d827881>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-14T05:00:13.174Z

Authors: Andrea Morandi, Mahesh Viswanathan

Published: 2026-05-12T14:22:06.000Z

## Abstract

\[Abridged\] Production LLM deployments receive feedback from a non-random fraction of users: thumbs sit mostly in the tails of the satisfaction distribution, and a naive average over them can land 40-50 percentage points away from true system quality. We treat this as a topic- and sentiment- stratified selection-bias problem and propose a three-agent hierarchical Bayesian pipeline that does not require ground-truth labels on individual interactions. A Topic Clustering Agent partitions the stream via UMAP + HDBSCAN over text embeddings; a Bias Modeling Agent fits a two-stage hierarchical Beta-Binomial under NUTS, inferring per-topic selection rates $s\_c$ and quality $q\_c$ with partial pooling; a Synthesis Agent reweights $q\_c$ by true topic prevalence $\\hat\\pi\_c = n\_c/N$ to report a bias-corrected aggregate posterior $\\bar Q = \\sum\_c \\hat\\pi\_c q\_c$ with credible interval, plus drift signals for online recalibration. Validation uses UltraFeedback (N=10,232 retained interactions, $C=18$ clusters, $Q^\\star=0.6249$) with simulated topic- and sentiment-dependent selection biases. We compare five Bayesian variants against Naive and IPW baselines. A mild prior on the feedback channel (typical positive-feedback rate and negative-to-positive ratio, both readable from any production dashboard without labels) keeps Hierarchical-Informed within 4-13 pp of $Q^\\star$ as the bias ratio sweeps from 1:1 to 30:1, with 95% credible intervals covering $Q^\\star$ in 50/50 random-seed replicates at $\\kappa\_{\\max}=10$. Without channel-side priors, every weak-prior variant misses $Q^\\star$ by 22-33 pp: the per-cluster sufficient statistics admit a one-parameter family of equally good fits, and the prior on the bias channel (not on latent quality) is what breaks the degeneracy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Aggregate Quality Recovery on UltraFeedback (kappa\_max = 10 Aggregate)](<https://www.sota2.com/research/sota/aggregate-quality-recovery-on-ultrafeedback-kappa_max-10-aggregate.md>): Aggregate Quality Recovery; UltraFeedback kappa\_max = 10 (Aggregate); Q-hat: 0.5484.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.12177>)

[PDF](<https://arxiv.org/pdf/2605.12177.pdf>)