# Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory

Source: [Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory](<https://www.sota2.com/research/paper/69e85ba17d0172140ec80f86>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-10T05:00:01.277Z

Authors: Masaki Uto

Published: 2026-04-21T06:29:13.000Z

## Abstract

Automated essay scoring (AES) is commonly evaluated on public benchmarks using quadratic weighted kappa (QWK). However, because benchmark labels are assigned by human raters and inevitably contain scoring errors, it remains unclear both what QWK is theoretically attainable and what level is practically sufficient for deployment. We therefore derive two dataset-specific QWK ceilings based on the reliability concept in classical test theory, which can be estimated from standard two-rater benchmarks without additional annotation. The first is the theoretical ceiling: the maximum QWK that an ideal AES model that perfectly predicts latent true scores can achieve under label noise. The second is the human-like ceiling: the QWK attainable by an AES model with human-level scoring error, providing a practical target when AES is intended to replace a single human rater. We further show that human--human QWK, often used as a ceiling reference, can underestimate the true ceiling. Simulation experiments validate the proposed ceilings, and experiments on real benchmarks illustrate how they clarify the current performance and remaining headroom of modern AES models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Automated Essay Scoring on ELLIPSE](<https://www.sota2.com/research/sota/automated-essay-scoring-on-ellipse.md>): Automated essay scoring; ELLIPSE; Quadratic Weighted Kappa: not recorded.

- [Automated Essay Scoring on ASAP Prompt 1 All essays (cross-validation)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-1-all-essays-cross-validation.md>): Automated essay scoring; ASAP Prompt 1 All essays (cross-validation); Quadratic Weighted Kappa (QWK): not recorded.

- [Automated Essay Scoring on ASAP Prompt 2 (val)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-2-val.md>): Automated essay scoring; ASAP Prompt 2 (val); QWK: not recorded.

- [Automated Essay Scoring on ASAP Prompt 3 All essays (val)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-3-all-essays-val.md>): Automated essay scoring; ASAP Prompt 3 All essays (val); Quadratic Weighted Kappa (QWK): not recorded.

- [Automated Essay Scoring on ASAP Prompt 4 (All essays cross-validation)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-4-all-essays-cross-validation.md>): Automated essay scoring; ASAP Prompt 4 (All essays cross-validation); QWK: not recorded.

- [Automated Essay Scoring on ASAP Prompt 5 All essays (cross-validation)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-5-all-essays-cross-validation.md>): Automated essay scoring; ASAP Prompt 5 All essays (cross-validation); Quadratic Weighted Kappa (QWK): not recorded.

- [Automated Essay Scoring on ASAP Prompt 6 (All essays val)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-6-all-essays-val.md>): Automated essay scoring; ASAP Prompt 6 (All essays val); Quadratic Weighted Kappa (QWK): not recorded.

- [Automated Essay Scoring on ASAP Prompt 7 (All essays cross-validation)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-7-all-essays-cross-validation.md>): Automated essay scoring; ASAP Prompt 7 (All essays cross-validation); QWK: not recorded.

- [Automated Essay Scoring on ASAP Prompt 8 (val)](<https://www.sota2.com/research/sota/automated-essay-scoring-on-asap-prompt-8-val.md>): Automated essay scoring; ASAP Prompt 8 (val); QWK: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.19131>)

[PDF](<https://arxiv.org/pdf/2604.19131.pdf>)
