# Line of Sight: On Linear Representations in VLLMs

Source: [Line of Sight: On Linear Representations in VLLMs](<https://www.sota2.com/research/paper/68fa79341e209131cea7ca2b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:54:52.275Z

Authors: Achyuta Rajaram, Sarah Schwettmann, Jacob Andreas, Arthur Conmy

Published: 2025-06-05T07:30:58.000Z

## Abstract

Language models can be equipped with multimodal capabilities by fine-tuning on embeddings of visual inputs. But how do such multimodal models represent images in their hidden activations? We explore representations of image concepts within LlaVA-Next, a popular open-source VLLM. We find a diverse set of ImageNet classes represented via linearly decodable features in the residual stream. We show that the features are causal by performing targeted edits on the model output. In order to increase the diversity of the studied linear features, we train multimodal Sparse Autoencoders (SAEs), creating a highly interpretable dictionary of text and image features. We find that although model representations across modalities are quite disjoint, they become increasingly shared in deeper layers.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.04706>)

[PDF](<https://arxiv.org/pdf/2506.04706.pdf>)