# Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat

Source: [Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat](<https://www.sota2.com/research/paper/68fa78c11e209131cea7a517>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:46:07.791Z

Authors: Pusheng Xu, Xia Gong, Xiaolan Chen, Weiyi Zhang, Jiancheng Yang, Bingjie Yan, Meng Yuan, Yalin Zheng, Mingguang He, Danli Shi

Published: 2025-05-26T07:45:42.000Z

## Abstract

Purpose: To develop a bilingual multimodal visual question answering (VQA) benchmark for evaluating VLMs in ophthalmology. Methods: Ophthalmic image posts and associated captions published between January 1, 2016, and December 31, 2024, were collected from WeChat Official Accounts. Based on these captions, bilingual question-answer (QA) pairs in Chinese and English were generated using GPT-4o-mini. QA pairs were categorized into six subsets by question type and language: binary (Binary\_CN, Binary\_EN), single-choice (Single-choice\_CN, Single-choice\_EN), and open-ended (Open-ended\_CN, Open-ended\_EN). The benchmark was used to evaluate the performance of three VLMs: GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B-Instruct. Results: The final OphthalWeChat dataset included 3,469 images and 30,120 QA pairs across 9 ophthalmic subspecialties, 548 conditions, 29 imaging modalities, and 68 modality combinations. Gemini 2.0 Flash achieved the highest overall accuracy (0.548), outperforming GPT-4o (0.522, P \< 0.001) and Qwen2.5-VL-72B-Instruct (0.514, P \< 0.001). It also led in both Chinese (0.546) and English subsets (0.550). Subset-specific performance showed Gemini 2.0 Flash excelled in Binary\_CN (0.687), Single-choice\_CN (0.666), and Single-choice\_EN (0.646), while GPT-4o ranked highest in Binary\_EN (0.717), Open-ended\_CN (BLEU-1: 0.301; BERTScore: 0.382), and Open-ended\_EN (BLEU-1: 0.183; BERTScore: 0.240). Conclusions: This study presents the first bilingual VQA benchmark for ophthalmology, distinguished by its real-world context and inclusion of multiple examinations per patient. The dataset reflects authentic clinical decision-making scenarios and enables quantitative evaluation of VLMs, supporting the development of accurate, specialized, and trustworthy AI systems for eye care.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.19624>)

[PDF](<https://arxiv.org/pdf/2505.19624.pdf>)