# Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain Question Answering

Source: [Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain Question Answering](<https://www.sota2.com/research/paper/68fa6e201e209131cea3dfcc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:50:11.375Z

Authors: Wanrong He, Andrew Mao, Jordan Boyd-Graber

Published: 2022-11-15T10:31:09.000Z

## Abstract

For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus. However, the strategies that computers use fundamentally differ from those of humans. To better understand these differences, we design a gamified interface for data collection -- Cheater's Bowl -- where a human answers complex questions with access to both traditional and modern search tools. We collect a dataset of human search sessions, analyze human search strategies, and compare them to state-of-the-art multi-hop QA models. Humans query logically, apply dynamic search chains, and use world knowledge to boost searching. We demonstrate how human queries can improve the accuracy of existing systems and propose improving the future design of QA models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2212.03296>)

[PDF](<https://arxiv.org/pdf/2212.03296.pdf>)