SAM 3: Segment Anything with Concepts
About
We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., "yellow school bus"), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.
Key insights
What problem does SAM 3 solve and how?
SAM 3 addresses Promptable Concept Segmentation: detecting, segmenting, and tracking all instances of a visual concept given as a noun phrase, image exemplar, or both, across images and videos. It uses a shared-backbone detector plus memory-based tracker, with a presence head decoupling recognition from localization.
How does SAM 3 differ from prior work?
Unlike SAM 2's single-instance visual prompting, SAM 3 finds all matching concept instances. On SA-Co/Gold it more than doubles the cgF1 of the strongest baseline OWLv2-star. A human-and-AI data engine produced 4M unique phrases with hard negatives across images and videos.
What are the practical uses of these findings?
SAM 3 supports interactive data annotation, robotics, and content creation, and can be combined with an MLLM for complex queries. The authors release checkpoints, code, and the SA-Co benchmark. On an H200 GPU, single-image inference runs in 30 ms; video latency scales with object count.
What limitations and future work do the authors note?
SAM 3 struggles with fine-grained out-of-domain concepts zero-shot. It is limited to simple noun phrases, not long referring expressions. Video inference cost scales linearly with tracked objects, and no shared global memory across objects exists yet.