# Does Your VFM Speak Plant? The Botanical Grammar of Vision Foundation Models for Object Detection

Source: [Does Your VFM Speak Plant? The Botanical Grammar of Vision Foundation Models for Object Detection](<https://www.sota2.com/research/paper/69ddc9e67d0172140ec797f1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-15T05:00:04.591Z

Authors: Lars Lundqvist, Earl Ranario, Hamid Kamangir, Heesup Yun, Christine Diepenbrock, Brian N. Bailey, J. Mason Earles

Published: 2026-04-10T21:30:45.000Z

## Abstract

Vision foundation models (VFMs) offer the promise of zero-shot object detection without task-specific training data, yet their performance in complex agricultural scenes remains highly sensitive to text prompt construction. We present a systematic prompt optimization framework evaluating four open-vocabulary detectors -- YOLO World, SAM3, Grounding DINO, and OWLv2 -- for cowpea flower and pod detection across synthetic and real field imagery. We decompose prompts into eight axes and conduct one-factor-at-a-time analysis followed by combinatorial optimization, revealing that models respond divergently to prompt structure: conditions that optimize one architecture can collapse another. Applying model-specific combinatorial prompts yields substantial gains over a naive species-name baseline, including +0.357 mAP@0.5 for YOLO World and +0.362 mAP@0.5 for OWLv2 on synthetic cowpea flower data. To evaluate cross-task generalization, we use an LLM to translate the discovered axis structure to a morphologically distinct target -- cowpea pods -- and compare against prompting using the discovered optimal structures from synthetic flower data. Crucially, prompt structures optimized exclusively on synthetic data transfer effectively to real-world fields: synthetic-pipeline prompts match or exceed those discovered on labeled real data for the majority of model-object combinations (flower: 0.374 vs. 0.353 for YOLO World; pod: 0.429 vs. 0.371 for SAM3). Our findings demonstrate that prompt engineering can substantially close the gap between zero-shot VFMs and supervised detectors without requiring manual annotation, and that optimal prompts are model-specific, non-obvious, and transferable across domains.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Object Detection on Cowpea Flower Synthetic](<https://www.sota2.com/research/sota/object-detection-on-cowpea-flower-synthetic.md>): Object Detection; Cowpea Flower Synthetic; mAP@0.5: not recorded.

- [Object Detection on Cowpea Flower Real](<https://www.sota2.com/research/sota/object-detection-on-cowpea-flower-real.md>): Object Detection; Cowpea Flower Real; mAP@0.5: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.09920>)

[PDF](<https://arxiv.org/pdf/2604.09920.pdf>)
