# Better Language Models Exhibit Higher Visual Alignment

Source: [Better Language Models Exhibit Higher Visual Alignment](<https://www.sota2.com/research/paper/68fa77061e209131cea70951>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-20T05:00:04.106Z

Authors: Jona Ruthardt, Gertjan J. Burghouts, Serge Belongie, Yuki M. Asano

Published: 2024-10-09T17:59:33.000Z

## Abstract

How well do text-only large language models (LLMs) align with the visual world? We present a systematic evaluation of this question by incorporating frozen representations of various language models into a discriminative vision-language framework and measuring zero-shot generalization to novel concepts. We find that decoder-based models exhibit stronger visual alignment than encoders, even when controlling for model and dataset size. Moreover, language modeling performance correlates with visual generalization, suggesting that advances in unimodal LLMs can simultaneously improve vision models. Leveraging these insights, we propose ShareLock, a lightweight method for fusing frozen vision and language backbones. ShareLock achieves robust performance across tasks while drastically reducing the need for paired data and compute. With just 563k image-caption pairs and under one GPU-hour of training, it reaches 51% accuracy on ImageNet. In cross-lingual settings, ShareLock dramatically outperforms CLIP, achieving 38.7% top-1 accuracy on Chinese image classification versus CLIP's 1.4%. Code is available.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CLIP Zero-shot Evaluation Suite (10 datasets)](<https://www.sota2.com/research/sota/image-classification-on-clip-zero-shot-evaluation-suite-10-datasets.md>): Image Classification; CLIP Zero-shot Evaluation Suite (10 datasets); Cars Accuracy: 13.2.

- [Compositional Reasoning on Winoground](<https://www.sota2.com/research/sota/compositional-reasoning-on-winoground.md>): Compositional Reasoning; Winoground; Group Score: 5.3.

- [Text-to-image retrieval on MSCOCO](<https://www.sota2.com/research/sota/text-to-image-retrieval-on-mscoco.md>): Text-to-Image Retrieval; MSCOCO; R@1: 16.5.

- [Image-to-text retrieval on MSCOCO](<https://www.sota2.com/research/sota/image-to-text-retrieval-on-mscoco.md>): Image-to-Text Retrieval; MSCOCO; R@1: 30.

- [Image-to-Text Retrieval on Flickr30k](<https://www.sota2.com/research/sota/image-to-text-retrieval-on-flickr30k.md>): Image-to-Text Retrieval; Flickr30K; R@1: 54.8.

- [Text-to-Image Retrieval on Flickr30K](<https://www.sota2.com/research/sota/text-to-image-retrieval-on-flickr30k.md>): Text-to-Image Retrieval; Flickr30K; R@1: 38.5.

- [Image Classification on ImageNet-1K (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-top-1-accuracy.md>): Image Classification; ImageNet-1K; Top-1 Acc: 59.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2410.07173>)

[PDF](<https://arxiv.org/pdf/2410.07173.pdf>)