# Looking Under the Hood : Tools for Diagnosing your Question Answering Engine

Source: [Looking Under the Hood : Tools for Diagnosing your Question Answering Engine](<https://www.sota2.com/research/paper/68fa5fa11e209131ce9eb23e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T05:45:11.378Z

Authors: Eric Breck, Marc Light, Gideon S. Mann, Ellen Riloff, Brianne Brown Pranav Anand, Mats Rooth, Michael Thelen

Published: 2001-07-03T18:06:05.000Z

## Abstract

In this paper we analyze two question answering tasks : the TREC-8 question answering task and a set of reading comprehension exams. First, we show that Q/A systems perform better when there are multiple answer opportunities per question. Next, we analyze common approaches to two subproblems: term overlap for answer sentence identification, and answer typing for short answer extraction. We present general tools for analyzing the strengths and limitations of techniques for these subproblems. Our results quantify the limitations of both term overlap and answer typing to distinguish between competing answer candidates.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/cs/0107006>)

[PDF](<https://arxiv.org/pdf/cs/0107006.pdf>)