# CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checks Source: [CROC: Evaluating and Training T2I Metrics with Pseudo- and Human-Labeled Contrastive Robustness Checks]() Collection index: [llms.txt]() Data updated: 2026-04-22T05:23:53.566Z Authors: Christoph Leiter, Yuki M. Asano, Margret Keuper, Steffen Eger Published: 2025-05-16T14:39:44.000Z ## Abstract The assessment of evaluation metrics (meta-evaluation) is crucial for determining the suitability of existing metrics in text-to-image (T2I) generation tasks. Human-based meta-evaluation is costly and time-intensive, and automated alternatives are scarce. We address this gap and propose CROC: a scalable framework for automated Contrastive Robustness Checks that systematically probes and quantifies metric robustness by synthesizing contrastive test cases across a comprehensive taxonomy of image properties. With CROC, we generate a pseudo-labeled dataset (CROC$^{syn}$) of over 1 million contrastive prompt-image pairs to enable a fine-grained comparison of evaluation metrics. We also use this dataset to train CROCScore, a new metric that achieves state-of-the-art performance among open-source methods, demonstrating an additional key application of our framework. To complement this dataset, we introduce a human-supervised benchmark (CROC$^{hum}$) targeting especially challenging categories. Our results highlight robustness issues in existing metrics: for example, many fail on prompts involving negation, and all tested open-source metrics fail on at least 24% of cases involving correct identification of body parts. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Text-to-Image Generation Evaluation on GenAi-Bench](): Text-to-image generation evaluation; GenAI-Bench; Kendall Tau B (Basic): 0.446. - [Text-to-Image Metric Meta-Evaluation on TIFA (Original)](): Text-to-Image Metric Meta-Evaluation; TIFA (Original); Kendall Correlation: 0.55. - [Text-to-Image Metric Meta-Evaluation on TIFA DSG](): Text-to-Image Metric Meta-Evaluation; TIFA DSG; Kendall Correlation: 0.538. - [Vision-Language Understanding on Winoground](): Vision-Language Understanding; Winoground; Text Accuracy: 61.5. - [T2I Metric Evaluation on CROCsyn Forward Text-Based](): T2I Metric Evaluation; CROCsyn Forward Text-Based; Scaled Accuracy: not recorded. - [T2I Metric Evaluation on CROCsyn Inverse Text-Based](): T2I Metric Evaluation; CROCsyn Inverse Text-Based; Scaled Accuracy: not recorded. - [T2I Metric Evaluation on CROCsyn Forward Image-Based](): T2I Metric Evaluation; CROCsyn Forward Image-Based; Scaled Accuracy: not recorded. - [T2I Metric Evaluation on CROCsyn Inverse Image-Based](): T2I Metric Evaluation; CROCsyn Inverse Image-Based; Scaled Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()