# VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework

Source: [VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework](<https://www.sota2.com/research/paper/68fa72861e209131cea5703b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:27:53.383Z

Authors: Chris Kelly, Luhui Hu, Bang Yang, Yu Tian, Deshun Yang, Cindy Yang, Zaoshan Huang, Zihao Li, Jiayin Hu, Yuexian Zou

Published: 2024-03-14T01:39:40.000Z

## Abstract

With the emergence of large language models (LLMs) and vision foundation models, how to combine the intelligence and capacity of these open-sourced or API-available models to achieve open-world visual perception remains an open question. In this paper, we introduce VisionGPT to consolidate and automate the integration of state-of-the-art foundation models, thereby facilitating vision-language understanding and the development of vision-oriented AI. VisionGPT builds upon a generalized multimodal framework that distinguishes itself through three key features: (1) utilizing LLMs (e.g., LLaMA-2) as the pivot to break down users' requests into detailed action proposals to call suitable foundation models; (2) integrating multi-source outputs from foundation models automatically and generating comprehensive responses for users; (3) adaptable to a wide range of applications such as text-conditioned image understanding/generation/editing and visual question answering. This paper outlines the architecture and capabilities of VisionGPT, demonstrating its potential to revolutionize the field of computer vision through enhanced efficiency, versatility, and generalization, and performance. Our code and models will be made publicly available. Keywords: VisionGPT, Open-world visual perception, Vision-language understanding, Large language model, and Foundation model

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Medical Report Generation on DeepEyeNet](<https://www.sota2.com/research/sota/medical-report-generation-on-deepeyenet.md>): Medical Report Generation; DeepEyeNet; BLEU-1: 0.353.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.09027>)

[PDF](<https://arxiv.org/pdf/2403.09027.pdf>)