# STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering

Source: [STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering](<https://www.sota2.com/research/paper/68fa75591e209131cea675bf>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:41:59.710Z

Authors: Guohao Sun, Can Qin, Huazhu Fu, Linwei Wang, Zhiqiang Tao

Published: 2024-06-28T15:01:23.000Z

## Abstract

Large Vision-Language Models (LVLMs) have shown significant potential in assisting medical diagnosis by leveraging extensive biomedical datasets. However, the advancement of medical image understanding and reasoning critically depends on building high-quality visual instruction data, which is costly and labor-intensive to obtain, particularly in the medical domain. To mitigate this data-starving issue, we introduce Self-Training Large Language and Vision Assistant for Medicine (STLLaVA-Med). The proposed method is designed to train a policy model (an LVLM) capable of auto-generating medical visual instruction data to improve data efficiency, guided through Direct Preference Optimization (DPO). Specifically, a more powerful and larger LVLM (e.g., GPT-4o) is involved as a biomedical expert to oversee the DPO fine-tuning process on the auto-generated data, encouraging the policy model to align efficiently with human preferences. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks, demonstrating competitive zero-shot performance with the utilization of only 9% of the medical data.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Medical Visual Question Answering on VQA-RAD](<https://www.sota2.com/research/sota/medical-visual-question-answering-on-vqa-rad.md>): Medical Visual Question Answering; VQA-RAD; Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2406.19973>)

[PDF](<https://arxiv.org/pdf/2406.19973.pdf>)