# LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs

Source: [LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs](<https://www.sota2.com/research/paper/68fa77f61e209131cea763fb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:29:27.078Z

Authors: Jiarui Wang, Huiyu Duan, Yu Zhao, Juntong Wang, Guangtao Zhai, Xiongkuo Min

Published: 2025-04-11T08:46:49.000Z

## Abstract

Recent breakthroughs in large multimodal models (LMMs) have significantly advanced both text-to-image (T2I) generation and image-to-text (I2T) interpretation. However, many generated images still suffer from issues related to perceptual quality and text-image alignment. Given the high cost and inefficiency of manual evaluation, an automatic metric that aligns with human preferences is desirable. To this end, we present EvalMi-50K, a comprehensive dataset and benchmark for evaluating large-multimodal image generation, which features (i) comprehensive tasks, encompassing 2,100 extensive prompts across 20 fine-grained task dimensions, and (ii) large-scale human-preference annotations, including 100K mean-opinion scores (MOSs) and 50K question-answering (QA) pairs annotated on 50,400 images generated from 24 T2I models. Based on EvalMi-50K, we propose LMM4LMM, an LMM-based metric for evaluating large multimodal T2I generation from multiple dimensions including perception, text-image correspondence, and task-specific accuracy. Extensive experimental results show that LMM4LMM achieves state-of-the-art performance on EvalMi-50K, and exhibits strong generalization ability on other AI-generated image evaluation benchmark datasets, manifesting the generality of both the EvalMi-50K dataset and LMM4LMM metric. Both EvalMi-50K and LMM4LMM will be released at https://github.com/IntMeGroup/LMM4LMM.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Semiosis Quality Evaluation on HGI SemiosisArt](<https://www.sota2.com/research/sota/semiosis-quality-evaluation-on-hgi-semiosisart.md>): Semiosis Quality Evaluation; HGI SemiosisArt; KRCC: 0.274.

- [Visual Question Answering on HGI SemiosisArt](<https://www.sota2.com/research/sota/visual-question-answering-on-hgi-semiosisart.md>): Visual Question Answering; HGI SemiosisArt; Accuracy: 44.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2504.08358>)

[PDF](<https://arxiv.org/pdf/2504.08358.pdf>)