# Teacher Network Calibration Improves Cross-Quality Knowledge Distillation

Source: [Teacher Network Calibration Improves Cross-Quality Knowledge Distillation](<https://www.sota2.com/research/paper/68fa6f151e209131cea43c7f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:13:37.852Z

Authors: Pia \\v{C}uk, Robin Senge, Mikko Lauri, Simone Frintrop

Published: 2023-04-15T16:55:43.000Z

## Abstract

We investigate cross-quality knowledge distillation (CQKD), a knowledge distillation method where knowledge from a teacher network trained with full-resolution images is transferred to a student network that takes as input low-resolution images. As image size is a deciding factor for the computational load of computer vision applications, CQKD notably reduces the requirements by only using the student network at inference time. Our experimental results show that CQKD outperforms supervised learning in large-scale image classification problems. We also highlight the importance of calibrating neural networks: we show that with higher temperature smoothing of the teacher's output distribution, the student distribution exhibits a higher entropy, which leads to both, a lower calibration error and a higher network accuracy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2304.07593>)

[PDF](<https://arxiv.org/pdf/2304.07593.pdf>)