Towards Accurate Pixel-wise Object Tracking by Attention Retrieval
About
The encoding of the target in object tracking moves from the coarse bounding-box to fine-grained segmentation map recently. Revisiting de facto real-time approaches that are capable of predicting mask during tracking, we observed that they usually fork a light branch from the backbone network for segmentation. Although efficient, directly fusing backbone features without considering the negative influence of background clutter tends to introduce false-negative predictions, lagging the segmentation accuracy. To mitigate this problem, we propose an attention retrieval network (ARN) to perform soft spatial constraints on backbone features. We first build a look-up-table (LUT) with the ground-truth mask in the starting frame, and then retrieves the LUT to obtain an attention map for spatial constraints. Moreover, we introduce a multi-resolution multi-stage segmentation network (MMS) to further weaken the influence of background clutter by reusing the predicted mask to filter backbone features. Our approach set a new state-of-the-art on recent pixel-wise object tracking benchmark VOT2020 while running at 40 fps. Notably, the proposed model surpasses SiamMask by 11.7/4.2/5.5 points on VOT2020, DAVIS2016, and DAVIS2017, respectively. We will release our code at https://github.com/researchmm/TracKit.
Key insights
Withdrawn preprint: the authors cited technical errors in September 2020. These notes describe the historical v2 manuscript, not a validated current result. Read the withdrawal notice
What problem did this paper address?
Pixel-wise trackers using lightweight segmentation branches suffered false-positive predictions from background clutter. The authors proposed an attention retrieval network (ARN) for soft spatial constraints and a multi-stage segmentation network (MMS) to filter features.
How did it differ from prior work?
The historical manuscript used a lookup table to retrieve spatial attention. It reported 0.444 expected average overlap on VOT2020 versus SiamMask's 0.328, with 40 FPS tracking; these claims belong to a preprint subsequently withdrawn for technical errors.
What were the practical findings?
The framework was a benchmark prototype evaluated on VOT2020, DAVIS2016, and DAVIS2017. It demonstrated real-time segmentation at 40 FPS and a cyclic initialization strategy for videos with only bounding-box initialization.
What limitations did the authors note?
The authors noted that top-ranked VOS methods generally surpassed VOT approaches and identified designing a unified model for both tasks as a research direction. They also observed memory errors in propagation-based methods on long sequences.