YOLOE: Real-Time Seeing Anything
About
Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with 3$\times$ less training cost and 1.4$\times$ inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 AP$^b$ and 0.4 AP$^m$ gains over closed-set YOLOv8-L with nearly 4$\times$ less training time. Code and models are available at https://github.com/THU-MIG/yoloe.
Key insights
What problem does YOLOE solve and how?
YOLOE extends YOLO detectors to recognize arbitrary objects via text, visual, or no prompts in one efficient model. It uses RepRTA for text alignment, SAVPE for visual cues, and LRPC for prompt-free retrieval.
How does YOLOE differ from prior work?
Unlike DINO-X's costly unified architecture, YOLOE-v8-S beats YOLO-Worldv2-S by 3.5 AP on LVIS with 3× less training and 1.4× faster inference on T4 GPU.
What are the practical uses of YOLOE?
YOLOE suits real-time open-vocabulary detection and segmentation on edge devices. The paper documents benchmark prototypes on T4 GPU and iPhone 12, not actual production deployments.
What limitations do the authors acknowledge?
Integrating detection and segmentation creates a multi-task trade-off, lowering frequent-category detection AP compared to detection-only models. Evaluation is limited to zero-shot LVIS and COCO transfer benchmarks.