MixFormer: End-to-End Tracking with Iterative Mixed Attention
About
Visual object tracking often employs a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, in this paper, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows to extract target-specific discriminative features and perform extensive communication between target and search area. Based on MAM, we build our MixFormer trackers simply by stacking multiple MAMs and placing a localization head on top. Specifically, we instantiate two types of MixFormer trackers, a hierarchical tracker MixCvT, and a non-hierarchical tracker MixViT. For these two trackers, we investigate a series of pre-training methods and uncover the different behaviors between supervised pre-training and self-supervised pre-training in our MixFormer trackers. We also extend the masked pre-training to our MixFormer trackers and design the competitive TrackMAE pre-training technique. Finally, to handle multiple target templates during online tracking, we devise an asymmetric attention scheme in MAM to reduce computational cost, and propose an effective score prediction module to select high-quality templates. Our MixFormer trackers set a new state-of-the-art performance on seven tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10k, OTB100 and UAV123. In particular, our MixViT-L achieves AUC score of 73.3% on LaSOT, 86.1% on TrackingNet, EAO of 0.584 on VOT2020, and AO of 75.7% on GOT-10k. Code and trained models are publicly available at https://github.com/MCG-NJU/MixFormer.
Key insights
What tracking problem does MixFormer solve?
Visual trackers typically use separate stages for feature extraction, target-search integration, and localization. MixFormer unifies extraction and integration via a Mixed Attention Module performing self-attention and cross-attention simultaneously on template and search tokens.
How does MixFormer differ from prior trackers?
Unlike CNN-backbone trackers with separate integration modules, MixFormer stacks MAMs as the backbone itself. MixViT-L achieves 73.3% AUC on LaSOT and 75.7% AO on GOT-10k, surpassing prior state-of-the-art trackers across seven benchmarks.
What are the practical uses of these findings?
The framework is evaluated on seven public tracking benchmarks including LaSOT, TrackingNet, and UAV123. The paper documents benchmark performance and VOT2022 challenge results but does not report any production deployment.
What limitations and future work do the authors note?
MixViT-L with ConvMAE pre-training obtains lower EAO on VOT2020 and lower AO on GOT-10k than with MAE, attributed to removing the last three ConvMAE layers. The authors state future work includes extending MixFormer to multiple object tracking or long-term tracking.