# Efficient Story Point Estimation With Comparative Learning

Source: [Efficient Story Point Estimation With Comparative Learning](<https://www.sota2.com/research/paper/68fa7a151e209131cea81016>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-18T05:00:18.507Z

Authors: Monoshiz Mahbub Khan, Xiaoyin Xi, Andrew Meneely, Yiming Tang, Zhe Yu

Published: 2025-07-19T14:36:19.000Z

## Abstract

Story points are unitless, project-specific effort estimates that help developers plan their sprints. Traditionally, developers have collaboratively estimated story points using planning poker or other manual techniques. Machine learning can reduce this burden, but only with sufficient context from the historical decisions made by the project team. That is, state-of-the-art models, such as GPT2SP and FastText-SVM, only make accurate (within-project) predictions when they are trained on data from the same project. The goal of this study is to streamline story point estimation by evaluating a comparative learning-based framework for calibrating project-specific story point prediction models. Instead of assigning a specific story point value to every backlog item, developers are presented with pairs of items and asked to indicate which item requires more effort. Using these comparative judgments, a machine learning model was trained to predict the story point estimates. We empirically evaluated our technique using data from 23,313 manual estimates across 16 projects. The model trained on comparative judgments achieved, on average, a 0.34 Spearman's rank correlation coefficient between its predictions and the ground truth story points. This is similar to, if not better than, the performance of a state-of-the-art regression model trained on ground truth story points. Through human subject experiments, the advantages of comparative judgments were validated - higher confidence, lower annotation time, and comparable agreement were observed for comparative judgments compared to direct ratings. In summary, the proposed comparative learning approach is more efficient than regression-based approaches, given its better performance, lower required annotation time, and higher training data reliability.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Story Point Estimation on appceleratorstudio](<https://www.sota2.com/research/sota/story-point-estimation-on-appceleratorstudio.md>): Story Point Estimation; appceleratorstudio; Spearman's Rho: 0.3254.

- [Story Point Estimation on aptanastudio](<https://www.sota2.com/research/sota/story-point-estimation-on-aptanastudio.md>): Story Point Estimation; aptanastudio; Rho: 0.3419.

- [Story Point Estimation on bamboo](<https://www.sota2.com/research/sota/story-point-estimation-on-bamboo.md>): Story Point Estimation; Bamboo; Spearman's Rho: 0.1768.

- [Story Point Estimation on clover](<https://www.sota2.com/research/sota/story-point-estimation-on-clover.md>): Story Point Estimation; clover; Rho: 0.4403.

- [Story Point Estimation on duracloud](<https://www.sota2.com/research/sota/story-point-estimation-on-duracloud.md>): Story Point Estimation; duracloud; Rho: 0.3758.

- [Story Point Estimation on jirasoftware](<https://www.sota2.com/research/sota/story-point-estimation-on-jirasoftware.md>): Story Point Estimation; jirasoftware; Spearman's Rho: 0.5324.

- [Story Point Estimation on mesos](<https://www.sota2.com/research/sota/story-point-estimation-on-mesos.md>): Story Point Estimation; mesos; Rho: 0.396.

- [Story Point Estimation on moodle](<https://www.sota2.com/research/sota/story-point-estimation-on-moodle.md>): Story Point Estimation; moodle; Spearman's Rho: 0.3499.

- [Story Point Estimation on mule](<https://www.sota2.com/research/sota/story-point-estimation-on-mule.md>): Story Point Estimation; mule; Spearman's Rho: 0.2342.

- [Story Point Estimation on springxd](<https://www.sota2.com/research/sota/story-point-estimation-on-springxd.md>): Story Point Estimation; springxd; Rho: 0.3982.

- [Story Point Estimation on titanium](<https://www.sota2.com/research/sota/story-point-estimation-on-titanium.md>): Story Point Estimation; titanium; Spearman's Rho: 0.1861.

- [Story Point Estimation on datamanagement](<https://www.sota2.com/research/sota/story-point-estimation-on-datamanagement.md>): Story Point Estimation; datamanagement; Spearman's Rho: 0.3775.

- [Story Point Estimation on mulestudio](<https://www.sota2.com/research/sota/story-point-estimation-on-mulestudio.md>): Story Point Estimation; mulestudio; Rho: 0.1096.

- [Story Point Estimation on talenddataquality](<https://www.sota2.com/research/sota/story-point-estimation-on-talenddataquality.md>): Story Point Estimation; talenddataquality; Spearman's Rho: 0.2892.

- [Story Point Estimation on talendesb](<https://www.sota2.com/research/sota/story-point-estimation-on-talendesb.md>): Story Point Estimation; talendesb; Spearman's Rho: 0.3453.

- [Story Point Estimation on usergrid](<https://www.sota2.com/research/sota/story-point-estimation-on-usergrid.md>): Story Point Estimation; usergrid; Rho: 0.2016.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.14642>)

[PDF](<https://arxiv.org/pdf/2507.14642.pdf>)
