# Vision Harnessing Agent for Open Ad-hoc Segmentation Source: [Vision Harnessing Agent for Open Ad-hoc Segmentation]() Collection index: [llms.txt]() Data updated: 2026-05-21T05:00:05.413Z Authors: Zilin Wang, Stella X. Yu Published: 2026-05-19T06:04:23.000Z ## Abstract Segmentation has become easy when the concept is known, requiring retrieval of a learned visual grounding from text. It remains hard for open ad-hoc concepts, where the grounding may not exist as one learned mask and must often be constructed from image evidence through parts, relations, exclusions, and collections. We propose a Vision-guided Ad-hoc Segmentation Agent (VASA), the first vision harnessing agent for open ad-hoc segmentation. VASA is training-free and couples a VLM agent, a segmentation foundation model, and a visually grounded workflow. Rather than revising text prompts alone, VASA uses a persistent working mask to reason, construct, and validate a solution. It plans visual operations, invokes segmentation tools, inspects results, edits the mask, and recovers from errors. We construct PARS, a new benchmark that turns part-level labels in PartImageNet into open ad-hoc concepts through long-form definition queries. On PARS, VASA outperforms open-vocabulary, reasoning-based, and agentic baselines, surpassing SAM3 Agent by 14-25%. On RefCOCOm, a standard multi-granularity referring segmentation benchmark, VASA improves over SAM3 Agent by 5-9% and over other agentic baselines by up to 20%. These results validate agentic visual construction for open ad-hoc segmentation. Our work points to a path for AI agents beyond wrapping foundation models as tools: Programming them with task knowledge, VLM behavior, visual routines, working memory, and failure-aware workflows. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Multi-granularity Referring Expression Segmentation on RefCOCOm (val)](): Multi-granularity Referring Expression Segmentation; RefCOCOm (val); gIoU (Part): 45. - [Multi-granularity Referring Expression Segmentation on RefCOCOm (testA)](): Multi-granularity Referring Expression Segmentation; RefCOCOm (testA); gIoU (Part): 43.2. - [Multi-granularity Referring Expression Segmentation on RefCOCOm (testB)](): Multi-granularity Referring Expression Segmentation; RefCOCOm (testB); gIoU (Part): 47.6. - [Open-vocabulary Part Segmentation on PARS Common Concepts](): Open-Vocabulary Part Segmentation; PARS Common Concepts; gIoU: 60.8. - [Open-vocabulary Part Segmentation on PARS Ad-hoc Concepts](): Open-Vocabulary Part Segmentation; PARS Ad-hoc Concepts; gIoU: 56.9. ## Original sources and implementation [Original paper]() [PDF]()