# Unsupervised Difficulty Estimation with Action Scores

Source: [Unsupervised Difficulty Estimation with Action Scores](<https://www.sota2.com/research/paper/68fa68c21e209131cea1e214>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:44:52.991Z

Authors: Octavio Arriaga, Matias Valdenegro-Toro

Published: 2020-11-23T15:18:44.000Z

## Abstract

Evaluating difficulty and biases in machine learning models has become of extreme importance as current models are now being applied in real-world situations. In this paper we present a simple method for calculating a difficulty score based on the accumulation of losses for each sample during training. We call this the action score. Our proposed method does not require any modification of the model neither any external supervision, as it can be implemented as callback that gathers information from the training process. We test and analyze our approach in two different settings: image classification, and object detection, and we show that in both settings the action score can provide insights about model and dataset biases.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2011.11461>)

[PDF](<https://arxiv.org/pdf/2011.11461.pdf>)