# Multi-Modality Task Cascade for 3D Object Detection Source: [Multi-Modality Task Cascade for 3D Object Detection]() Collection index: [llms.txt]() Data updated: 2025-10-25T23:23:37.152Z Authors: Jinhyung Park, Xinshuo Weng, Yunze Man, Kris Kitani Published: 2021-07-08T17:55:01.000Z ## Abstract Point clouds and RGB images are naturally complementary modalities for 3D visual understanding - the former provides sparse but accurate locations of points on objects, while the latter contains dense color and texture information. Despite this potential for close sensor fusion, many methods train two models in isolation and use simple feature concatenation to represent 3D sensor data. This separated training scheme results in potentially sub-optimal performance and prevents 3D tasks from being used to benefit 2D tasks that are often useful on their own. To provide a more integrated approach, we propose a novel Multi-Modality Task Cascade network (MTC-RCNN) that leverages 3D box proposals to improve 2D segmentation predictions, which are then used to further refine the 3D boxes. We show that including a 2D network between two stages of 3D modules significantly improves both 2D and 3D task performance. Moreover, to prevent the 3D module from over-relying on the overfitted 2D predictions, we propose a dual-head 2D segmentation training and inference scheme, allowing the 2nd 3D module to learn to interpret imperfect 2D segmentation predictions. Evaluating our model on the challenging SUN RGB-D dataset, we improve upon state-of-the-art results of both single modality and fusion networks by a large margin ($\\textbf{+3.8}$ mAP@0.5). Code will be released $\\href{https://github.com/Divadi/MTC\_RCNN}{\\text{here.}}$ ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [3D Object Detection on SUN RGB-D (mAP@0.25, mAP@0.5, FPS)](): 3D Object Detection; SUN RGB-D; mAP@0.25: 65.3. - [3D Object Detection on SUN RGB-D (val)](): 3D Object Detection; SUN RGB-D (val); mAP@0.25: 65.3. ## Original sources and implementation [Original paper]() [PDF]()