# When Negation Is a Geometry Problem in Vision-Language Models

Source: [When Negation Is a Geometry Problem in Vision-Language Models](<https://www.sota2.com/research/paper/69c21a5c0636e5bf4afc45ce>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-07T05:00:04.607Z

Authors: Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov, Nikos Deligiannis

Published: 2026-03-20T23:06:23.000Z

## Abstract

Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image-Text Retrieval on N-COCO](<https://www.sota2.com/research/sota/image-text-retrieval-on-n-coco.md>): Image-Text Retrieval; N-COCO; R@1: 80.

- [Negation Understanding on SimpleNeg COCO 2014 1.0 (val)](<https://www.sota2.com/research/sota/negation-understanding-on-simpleneg-coco-2014-1-0-val.md>): Negation Understanding; SimpleNeg COCO 2014 1.0 (val); Qwen3-VL-32B Top-1 Accuracy: 54.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.20554>)

[PDF](<https://arxiv.org/pdf/2603.20554.pdf>)