# Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Experts

Source: [Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Experts](<https://www.sota2.com/research/paper/68fa6ff81e209131cea483e5>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:30:17.231Z

Authors: Johannes Jakubik, Daniel Weber, Patrick Hemmer, Michael V\\"ossing, Gerhard Satzger

Published: 2023-07-06T14:06:23.000Z

## Abstract

Information systems increasingly leverage artificial intelligence (AI) and machine learning (ML) to generate value from vast amounts of data. However, ML models are imperfect and can generate incorrect classifications. Hence, human-in-the-loop (HITL) extensions to ML models add a human review for instances that are difficult to classify. This study argues that continuously relying on human experts to handle difficult model classifications leads to a strong increase in human effort, which strains limited resources. To address this issue, we propose a hybrid system that creates artificial experts that learn to classify data instances from unknown classes previously reviewed by human experts. Our hybrid system assesses which artificial expert is suitable for classifying an instance from an unknown class and automatically assigns it. Over time, this reduces human effort and increases the efficiency of the system. Our experiments demonstrate that our approach outperforms traditional HITL systems for several benchmarks on image classification.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2307.03003>)

[PDF](<https://arxiv.org/pdf/2307.03003.pdf>)
