# Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models

Source: [Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models](<https://www.sota2.com/research/paper/68fa74481e209131cea60bca>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:37:02.192Z

Authors: Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino, Richard E Turner

Published: 2024-07-23T14:39:40.000Z

## Abstract

Vision language models (VLMs) demonstrate impressive capabilities in visual question answering and image captioning, acting as a crucial link between visual and language models. However, existing open-source VLMs heavily rely on pretrained and frozen vision encoders (such as CLIP). Despite CLIP's robustness across diverse domains, it still exhibits non-negligible image understanding errors. These errors propagate to the VLM responses, resulting in sub-optimal performance. In our work, we propose an efficient and robust method for updating vision encoders within VLMs. Our approach selectively and locally updates encoders, leading to substantial performance improvements on data where previous mistakes occurred, while maintaining overall robustness. Furthermore, we demonstrate the effectiveness of our method during continual few-shot updates. Theoretical grounding, generality, and computational efficiency characterize our approach.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2407.16526>)

[PDF](<https://arxiv.org/pdf/2407.16526.pdf>)
