# GalleryGPT: Analyzing Paintings with Large Multimodal Models

Source: [GalleryGPT: Analyzing Paintings with Large Multimodal Models](<https://www.sota2.com/research/paper/68fa744f1e209131cea60f40>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:06:43.001Z

Authors: Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, Heng Tao Shen

Published: 2024-08-01T11:52:56.000Z

## Abstract

Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, i.e., painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. \\textcolor{blue}{The codes and model are available at: https://github.com/steven640pixel/GalleryGPT.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Artist Influence Prediction on WIB-100 balanced](<https://www.sota2.com/research/sota/artist-influence-prediction-on-wib-100-balanced.md>): Artist Influence Prediction; WIB-100 balanced; Precision: 65.1.

- [Visual Question Answering on HGI SemiosisArt](<https://www.sota2.com/research/sota/visual-question-answering-on-hgi-semiosisart.md>): Visual Question Answering; HGI SemiosisArt; Accuracy: 26.1.

- [Semiosis Quality Evaluation on HGI SemiosisArt](<https://www.sota2.com/research/sota/semiosis-quality-evaluation-on-hgi-semiosisart.md>): Semiosis Quality Evaluation; HGI SemiosisArt; KRCC: -0.034.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2408.00491>)

[PDF](<https://arxiv.org/pdf/2408.00491.pdf>)