# Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection

Source: [Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection](<https://www.sota2.com/research/paper/69e704c67d0172140ec7ef3d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-22T05:21:28.944Z

Authors: Dawar Jyoti Deka, Amit Sethi, Syed Mohammad Ali

Published: 2026-04-18T19:55:05.000Z

## Abstract

Vision-language models enable open-vocabulary object grounding through natural language queries, under the implicit assumption that semantically equivalent descriptions yield consistent outputs. We examine this assumption using a controlled pipeline combining DETR for object proposals with CLIP for language-conditioned selection on 263 COCO val2017 images. We find that overlapping prompts such as "a person," "a human," and "a pedestrian" frequently select different instances, with mean instability of 2.11 distinct selections across six prompts. PCA analysis shows this variability is structured and directional, not random. Prompt ensembling does not improve quality and often shifts selections toward generic regions. We further show that text embedding proximity explains only 34% of grounding disagreement (r = -0.58), confirming that instability arises from the argmax selection mechanism rather than text-level distances alone.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.17126>)

[PDF](<https://arxiv.org/pdf/2604.17126.pdf>)