# Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)

Source: [Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)](<https://www.sota2.com/research/paper/68fa757a1e209131cea683c9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:41:59.710Z

Authors: Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, William Yang Wang

Published: 2024-04-05T17:57:16.000Z

## Abstract

With advances in the quality of text-to-image (T2I) models has come interest in benchmarking their prompt faithfulness -- the semantic coherence of generated images to the prompts they were conditioned on. A variety of T2I faithfulness metrics have been proposed, leveraging advances in cross-modal embeddings and vision-language models (VLMs). However, these metrics are not rigorously compared and benchmarked, instead presented with correlation to human Likert scores over a set of easy-to-discriminate images against seemingly weak baselines.   We introduce T2IScoreScore, a curated set of semantic error graphs containing a prompt and a set of increasingly erroneous images. These allow us to rigorously judge whether a given prompt faithfulness metric can correctly order images with respect to their objective error count and significantly discriminate between different error nodes, using meta-metric scores derived from established statistical tests. Surprisingly, we find that the state-of-the-art VLM-based metrics (e.g., TIFA, DSG, LLMScore, VIEScore) we tested fail to significantly outperform simple (and supposedly worse) feature-based metrics like CLIPScore, particularly on a hard subset of naturally-occurring T2I model errors. TS2 will enable the development of better T2I prompt faithfulness metrics through more rigorous comparison of their conformity to expected orderings and separations under objective criteria.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [T2I Metric Evaluation on CROCsyn Inverse Text-Based](<https://www.sota2.com/research/sota/t2i-metric-evaluation-on-crocsyn-inverse-text-based.md>): T2I Metric Evaluation; CROCsyn Inverse Text-Based; Scaled Accuracy: 0.904.

- [T2I Metric Evaluation on CROCsyn Forward Text-Based](<https://www.sota2.com/research/sota/t2i-metric-evaluation-on-crocsyn-forward-text-based.md>): T2I Metric Evaluation; CROCsyn Forward Text-Based; Scaled Accuracy: 0.922.

- [T2I Metric Evaluation on CROCsyn Forward Image-Based](<https://www.sota2.com/research/sota/t2i-metric-evaluation-on-crocsyn-forward-image-based.md>): T2I Metric Evaluation; CROCsyn Forward Image-Based; Scaled Accuracy: 95.8.

- [T2I Metric Evaluation on CROCsyn Inverse Image-Based](<https://www.sota2.com/research/sota/t2i-metric-evaluation-on-crocsyn-inverse-image-based.md>): T2I Metric Evaluation; CROCsyn Inverse Image-Based; Scaled Accuracy: 97.9.

- [Text-to-Image Generation Evaluation on GenAi-Bench](<https://www.sota2.com/research/sota/text-to-image-generation-evaluation-on-genai-bench.md>): Text-to-image generation evaluation; GenAI-Bench; Kendall Tau B (Basic): 0.072.

- [Vision-Language Understanding on Winoground](<https://www.sota2.com/research/sota/vision-language-understanding-on-winoground.md>): Vision-Language Understanding; Winoground; Text Accuracy: 35.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.04251>)

[PDF](<https://arxiv.org/pdf/2404.04251.pdf>)