# An Automatic Question Usability Evaluation Toolkit

Source: [An Automatic Question Usability Evaluation Toolkit](<https://www.sota2.com/research/paper/68fa73621e209131cea5c19d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:48:16.237Z

Authors: Steven Moore, Eamon Costello, Huy A. Nguyen, John Stamper

Published: 2024-05-30T23:04:53.000Z

## Abstract

Evaluating multiple-choice questions (MCQs) involves either labor intensive human assessments or automated methods that prioritize readability, often overlooking deeper question design flaws. To address this issue, we introduce the Scalable Automatic Question Usability Evaluation Toolkit (SAQUET), an open-source tool that leverages the Item-Writing Flaws (IWF) rubric for a comprehensive and automated quality evaluation of MCQs. By harnessing the latest in large language models such as GPT-4, advanced word embeddings, and Transformers designed to analyze textual complexity, SAQUET effectively pinpoints and assesses a wide array of flaws in MCQs. We first demonstrate the discrepancy between commonly used automated evaluation metrics and the human assessment of MCQ quality. Then we evaluate SAQUET on a diverse dataset of MCQs across the five domains of Chemistry, Statistics, Computer Science, Humanities, and Healthcare, showing how it effectively distinguishes between flawed and flawless questions, providing a level of analysis beyond what is achievable with traditional metrics. With an accuracy rate of over 94% in detecting the presence of flaws identified by human evaluators, our findings emphasize the limitations of existing evaluation methods and showcase potential in improving the quality of educational assessments.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Writing flaw detection on BenchMarker Writing - Out of Domain, Human](<https://www.sota2.com/research/sota/writing-flaw-detection-on-benchmarker-writing-out-of-domain-human.md>): Writing flaw detection; BenchMarker Writing - Out of Domain, Human; Accuracy: 92.6.

- [Writing flaw detection on BenchMarker Writing - In Domain, NLP](<https://www.sota2.com/research/sota/writing-flaw-detection-on-benchmarker-writing-in-domain-nlp.md>): Writing flaw detection; BenchMarker Writing - In Domain, NLP; Accuracy: 77.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.20529>)

[PDF](<https://arxiv.org/pdf/2405.20529.pdf>)