# torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation

Source: [torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation](<https://www.sota2.com/research/paper/68fa6b471e209131cea2d3d8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:23:55.205Z

Authors: Yoshitomo Matsubara

Published: 2020-11-25T17:51:30.000Z

## Abstract

While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work. Such frameworks, however, are usually neither well generalized nor maintained, thus researchers are still required to write a lot of code to refactor/build on the frameworks for introducing new methods, models, datasets and designing experiments. In this paper, we present our developed open-source framework built on PyTorch and dedicated for knowledge distillation studies. The framework is designed to enable users to design experiments by declarative PyYAML configuration files, and helps researchers complete the recently proposed ML Code Completeness Checklist. Using the developed framework, we demonstrate its various efficient training strategies, and implement a variety of knowledge distillation methods. We also reproduce some of their original experimental results on the ImageNet and COCO datasets presented at major machine learning conferences such as ICLR, NeurIPS, CVPR and ECCV, including recent state-of-the-art methods. All the source code, configurations, log files and trained model weights are publicly available at https://github.com/yoshitomo-matsubara/torchdistill .

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Object Detection on COCO 2017 (val)](<https://www.sota2.com/research/sota/object-detection-on-coco-2017-val.md>): Object Detection; COCO 2017 (val); AP: not recorded.

- [Instance Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017-val.md>): Instance Segmentation; COCO 2017 (val); APm: not recorded.

- [Image Classification on ILSVRC 2012 (val)](<https://www.sota2.com/research/sota/image-classification-on-ilsvrc-2012-val.md>): Image Classification; ILSVRC 2012 (val); Top-1 Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2011.12913>)

[PDF](<https://arxiv.org/pdf/2011.12913.pdf>)

[Implementation](<https://github.com/yoshitomo-matsubara/torchdistill>)
