# Structured Knowledge Distillation for Dense Prediction Source: [Structured Knowledge Distillation for Dense Prediction]() Collection index: [llms.txt]() Data updated: 2025-10-27T06:02:46.151Z Authors: Yifan Liu, Changyong Shun, Jingdong Wang, Chunhua Shen Published: 2019-03-11T10:05:09.000Z ## Abstract In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at: https://git.io/StructKD ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [3D Semantic Segmentation on ScanNet v2 (val)](): 3D Semantic Segmentation; ScanNet V2 (val); mIoU: 56.36. - [Semantic Segmentation on Cityscapes (val) (mIoU)](): Semantic segmentation; Cityscapes (val); mIoU: 69.1. ## Original sources and implementation [Original paper]() [PDF]()