# A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models

Source: [A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models](<https://www.sota2.com/research/paper/68fa76f41e209131cea70396>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:07:06.968Z

Authors: Xiujie Song, Mengyue Wu, Kenny Q. Zhu, Chunhao Zhang, Yanyi Chen

Published: 2024-02-28T15:28:36.000Z

## Abstract

Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the Cookie Theft task in human cognitive tests, we propose a novel evaluation benchmark to evaluate high-level cognitive abilities of LVLMs using images with rich semantics. The benchmark consists of 251 images along with comprehensive annotations. It defines eight reasoning capabilities and comprises an image description task and a visual question answering task. Our evaluation of well-known LVLMs shows that there is still a significant gap in cognitive abilities between LVLMs and humans.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Storytelling Image Generation on CogBench ISA (test)](<https://www.sota2.com/research/sota/storytelling-image-generation-on-cogbench-isa-test.md>): Storytelling Image Generation; CogBench ISA (test); Story Length (Words): 135.93.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2402.18409>)

[PDF](<https://arxiv.org/pdf/2402.18409.pdf>)