# HICEScore: A Hierarchical Metric for Image Captioning Evaluation Source: [HICEScore: A Hierarchical Metric for Image Captioning Evaluation]() Collection index: [llms.txt]() Data updated: 2025-10-27T06:37:02.192Z Authors: Zequn Zeng, Jianqiao Sun, Hao Zhang, Tiansheng Wen, Yudi Su, Yan Xie, Zhengjue Wang, Bo Chen Published: 2024-07-26T08:24:30.000Z ## Abstract Image captioning evaluation metrics can be divided into two categories, reference-based metrics and reference-free metrics. However, reference-based approaches may struggle to evaluate descriptive captions with abundant visual details produced by advanced multimodal large language models, due to their heavy reliance on limited human-annotated references. In contrast, previous reference-free metrics have been proven effective via CLIP cross-modality similarity. Nonetheless, CLIP-based metrics, constrained by their solution of global image-text compatibility, often have a deficiency in detecting local textual hallucinations and are insensitive to small visual objects. Besides, their single-scale designs are unable to provide an interpretable evaluation process such as pinpointing the position of caption mistakes and identifying visual regions that have not been described. To move forward, we propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S). By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism, breaking through the barriers of the single-scale structure of existing reference-free metrics. Comprehensive experiments indicate that our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics like CLIP-S and PAC-S, and reference-based metrics like METEOR and CIDEr. Moreover, several case studies reveal that the assessment process of HICE-S on detailed captions closely resembles interpretable human judgments.Our code is available at https://github.com/joeyz0z/HICE. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Correlation with human judgments on Flickr8k Expert (tau\_b)](): Correlation with Human Judgments; Flickr8k Expert; Tau-b Correlation: 57.2. - [Image-Text Alignment Evaluation on Flickr8k Expert 36 (test)](): Image-Text Alignment Evaluation; Flickr8k Expert 36 (test); Tau-c: 56.4. - [Image-Text Alignment Evaluation on Flickr8k CrowdFlower 36](): Image-Text Alignment Evaluation; Flickr8k CrowdFlower 36; Kendall's Tau\_b: 37.2. - [Hallucination Detection on FOIL (Reference-Conditioned Accuracy)](): Hallucination Detection; FOIL; Accuracy (4 Refs): 97. - [Image-Text Alignment Evaluation on Pascal-50S 14 (test)](): Image-Text Alignment Evaluation; Pascal-50S 14 (test); HC: 68.6. - [Image-Text Alignment Evaluation on Composite 37 (test)](): Image-Text Alignment Evaluation; Composite 37 (test); Kendall's Tau-c: 57.9. - [Image Captioning Evaluation on Pascal-50S](): Image Captioning Evaluation; Pascal-50S; Accuracy: 86.1. - [Correlation with Human Judgment on Flickr8K-CF](): Correlation with human judgment; Flickr8K-CF; Tau B: 38.2. - [Image Captioning Evaluation on FOIL](): Image Captioning Evaluation; FOIL; Accuracy (4-ref): 97. - [Image Captioning Evaluation on Flickr8k Expert](): Image Captioning Evaluation; Flickr8k Expert; Kendall Tau-c (tau\_c): 57.7. - [Image Captioning Evaluation on Flickr8k-CF](): Image Captioning Evaluation; Flickr8K-CF; Kendall-b Correlation (tau\_b): 38.2. - [Image Captioning Evaluation on Composite](): Image Captioning Evaluation; Composite; Kendall-c Tau\_c: 58.7. ## Original sources and implementation [Original paper]() [PDF]()