# A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation

Source: [A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation](<https://www.sota2.com/research/paper/68fa67c91e209131cea18623>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:22:05.587Z

Authors: Jan Deriu, Katsiaryna Mlynchyk, Philippe Schl\\"apfer, Alvaro Rodrigo, Dirk von Gr\\"unigen, Nicolas Kaiser, Kurt Stockinger, Eneko Agirre, Mark Cieliebak

Published: 2020-04-16T12:50:01.000Z

## Abstract

In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database called Operation Trees (OT). This representation allows us to invert the annotation process without losing flexibility in the types of queries that we generate. Furthermore, it allows for fine-grained alignment of query tokens to OT operations. In our method, we randomly generate OTs from a context-free grammar. Afterwards, annotators have to write the appropriate natural language question that is represented by the OT. Finally, the annotators assign the tokens to the OT operations. We apply the method to create a new corpus OTTA (Operation Trees and Token Assignment), a large semantic parsing corpus for evaluating natural language interfaces to databases. We compare OTTA to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art semantic parsing model on our data and show that our corpus is a challenging dataset and that the token alignment can be leveraged to increase the performance significantly.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2004.07633>)

[PDF](<https://arxiv.org/pdf/2004.07633.pdf>)