# Pixel-Wise Contrastive Distillation Source: [Pixel-Wise Contrastive Distillation]() Collection index: [llms.txt]() Data updated: 2025-10-27T06:32:05.519Z Authors: Junqiang Huang, Zichao Guo Published: 2022-11-01T02:00:32.000Z ## Abstract We present a simple but effective pixel-level self-supervised distillation framework friendly to dense prediction tasks. Our method, called Pixel-Wise Contrastive Distillation (PCD), distills knowledge by attracting the corresponding pixels from student's and teacher's output feature maps. PCD includes a novel design called SpatialAdaptor which \`\`reshapes'' a part of the teacher network while preserving the distribution of its output features. Our ablation experiments suggest that this reshaping behavior enables more informative pixel-to-pixel distillation. Moreover, we utilize a plug-in multi-head self-attention module that explicitly relates the pixels of student's feature maps to enhance the effective receptive field, leading to a more competitive student. PCD \\textbf{outperforms} previous self-supervised distillation methods on various dense prediction tasks. A backbone of \\mbox{ResNet-18-FPN} distilled by PCD achieves $37.4$ AP$^\\text{bbox}$ and $34.0$ AP$^\\text{mask}$ on COCO dataset using the detector of \\mbox{Mask R-CNN}. We hope our study will inspire future research on how to pre-train a small model friendly to dense prediction tasks in a self-supervised fashion. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()