# DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation from Stereo Imagery Source: [DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation from Stereo Imagery]() Collection index: [llms.txt]() Data updated: 2025-10-25T21:34:37.515Z Authors: Junming Zhang, Katherine A. Skinner, Ram Vasudevan, Matthew Johnson-Roberson Published: 2018-09-13T01:36:55.000Z ## Abstract Recent work has shown that convolutional neural networks (CNNs) can be applied successfully in disparity estimation, but these methods still suffer from errors in regions of low-texture, occlusions and reflections. Concurrently, deep learning for semantic segmentation has shown great progress in recent years. In this paper, we design a CNN architecture that combines these two tasks to improve the quality and accuracy of disparity estimation with the help of semantic segmentation. Specifically, we propose a network structure in which these two tasks are highly coupled. One key novelty of this approach is the two-stage refinement process. Initial disparity estimates are refined with an embedding learned from the semantic segmentation branch of the network. The proposed model is trained using an unsupervised approach, in which images from one half of the stereo pair are warped and compared against images from the other camera. Another key advantage of the proposed approach is that a single network is capable of outputting disparity estimates and semantic labels. These outputs are of great use in autonomous vehicle operation; with real-time constraints being key, such performance improvements increase the viability of driving applications. Experiments on KITTI and Cityscapes datasets show that our model can achieve state-of-the-art results and that leveraging embedding learned from semantic segmentation improves the performance of disparity estimation. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Stereo Matching on KITTI 2015 (test)](): Stereo Matching; KITTI 2015 (test); D1 Error (Overall): 6.33. - [Stereo Matching on KITTI 2012 (test)](): Stereo Matching; KITTI 2012 (test); Outlier Rate (3px, Noc): not recorded. ## Original sources and implementation [Original paper]() [PDF]()