# CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs Source: [CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:11:37.234Z Authors: Yassine Ouali, Adrian Bulat, Brais Martinez, Georgios Tzimiropoulos Published: 2024-08-19T21:56:20.000Z ## Abstract Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Science Question Answering on ScienceQA (Image Acc.)](): Science Question Answering; ScienceQA; Image Accuracy: 67.6. - [Object Hallucination Mitigation on AMBER](): Object Hallucination Mitigation on Generative Tasks; AMBER; CHAIR: 3.7. - [Hallucination Evaluation on AMBER](): Hallucination Evaluation; AMBER; CHAIR: 3.7. - [Visual Question Answering on TextVQA](): Visual Question Answering; TextVQA; Accuracy: 56.4. ## Original sources and implementation [Original paper]() [PDF]()