# General-Purpose Question-Answering with Macaw

Source: [General-Purpose Question-Answering with Macaw](<https://www.sota2.com/research/paper/68fa6ab91e209131cea2a13d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:32:17.810Z

Authors: Oyvind Tafjord, Peter Clark

Published: 2021-09-06T16:37:46.000Z

## Abstract

Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available. In response, we present Macaw, a versatile, generative question-answering (QA) system that we are making available to the community. Macaw is built on UnifiedQA, itself built on T5, and exhibits strong performance, zero-shot, on a wide variety of topics, including outperforming GPT-3 by over 10% (absolute) on Challenge300, a suite of 300 challenge questions, despite being an order of magnitude smaller (11 billion vs. 175 billion parameters). In addition, Macaw allows different permutations ("angles") of its inputs and outputs to be used, for example Macaw can take a question and produce an answer; or take an answer and produce a question; or take an answer and question, and produce multiple-choice options. We describe the system, and illustrate a variety of question types where it produces surprisingly good answers, well outside the training setup. We also identify question classes where it still appears to struggle, offering insights into the limitations of pretrained language models. Macaw is freely available, and we hope that it proves useful to the community. Macaw is available at https://github.com/allenai/macaw

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Parts Mental Model Accuracy on ParRoT++](<https://www.sota2.com/research/sota/parts-mental-model-accuracy-on-parrot-plus-plus.md>): Parts Mental Model Accuracy; ParRoT++; Base LM Accuracy: 60.04.

- [Parts mental model prediction on ParRoT](<https://www.sota2.com/research/sota/parts-mental-model-prediction-on-parrot.md>): Parts mental model prediction; ParRoT; Accuracy: 0.5945.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2109.02593>)

[PDF](<https://arxiv.org/pdf/2109.02593.pdf>)