# Quantifying the Impact of Cognitive Biases in Question-Answering Systems

Source: [Quantifying the Impact of Cognitive Biases in Question-Answering Systems](<https://www.sota2.com/research/paper/68fa66731e209131cea10265>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:49:28.917Z

Authors: Keith Burghardt, Tad Hogg, Kristina Lerman

Published: 2019-09-20T17:54:49.000Z

## Abstract

Crowdsourcing can identify high-quality solutions to problems; however, individual decisions are constrained by cognitive biases. We investigate some of these biases in an experimental model of a question-answering system. In both natural and controlled experiments, we observe a strong position bias in favor of answers appearing earlier in a list of choices. This effect is enhanced by three cognitive factors: the attention an answer receives, its perceived popularity, and cognitive load, measured by the number of choices a user has to process. While separately weak, these effects synergistically amplify position bias and decouple user choices of best answers from their intrinsic quality. We end our paper by discussing the novel ways we can apply these findings to substantially improve how high-quality answers are found in question-answering systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1909.09633>)

[PDF](<https://arxiv.org/pdf/1909.09633.pdf>)