# Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean

Source: [Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean](<https://www.sota2.com/research/paper/69dc788a7d0172140ec78aa8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-14T05:00:04.234Z

Authors: Aleksandr Meshkov

Published: 2026-04-04T09:04:38.000Z

## Abstract

Existing evaluation methods for LLM-based AI systems, such as LLM-as-a-Judge, verdict systems, and NLI, do not always align well with human assessment because they cannot adapt their strictness to the application domain. This paper presents Temperature-Controlled Verdict Aggregation (TCVA), a method that combines a five-level verdict-scoring system with generalized power-mean aggregation and an intuitive temperature parameter T \[0.1, 1.0\] to control evaluation rigor. Low temperatures yield pessimistic scores suited for safety-critical domains; high temperatures produce lenient scores appropriate for conversational AI. Experimental evaluation on three benchmark datasets with human Likert-scale annotations (SummEval and USR) shows that TCVA achieves correlation with human judgments comparable to RAGAS on faithfulness (Spearman = 0.667 vs. 0.676) while consistently outperforming DeepEval. The method requires no additional LLM calls when adjusting the temperature parameter.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Relevancy Assessment on SummEval Rel](<https://www.sota2.com/research/sota/relevancy-assessment-on-summeval-rel.md>): Relevancy; SummEval Rel; Spearman's Rho: 0.48.

- [Dialogue on USR (N = 198)](<https://www.sota2.com/research/sota/dialogue-on-usr-n-198.md>): Dialogue; USR (N = 198); Spearman's Rho: 0.173.

- [Dialogue Evaluation on USR](<https://www.sota2.com/research/sota/dialogue-evaluation-on-usr.md>): Dialogue Evaluation; USR; Spearman's rho: 0.173.

- [Dialogue Faithfulness Evaluation on USR](<https://www.sota2.com/research/sota/dialogue-faithfulness-evaluation-on-usr.md>): Dialogue Faithfulness Evaluation; USR; Kendall's Tau: 0.143.

- [Faithfulness Evaluation on SummEval](<https://www.sota2.com/research/sota/faithfulness-evaluation-on-summeval.md>): Faithfulness Evaluation; SummEval; Kendall's tau (τ): 0.527.

- [Faithfulness on SummEval](<https://www.sota2.com/research/sota/faithfulness-on-summeval.md>): Faithfulness; SummEval; Spearman's Rho: 0.667.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.08595>)

[PDF](<https://arxiv.org/pdf/2604.08595.pdf>)