# ViTKD: Practical Guidelines for ViT feature knowledge distillation Source: [ViTKD: Practical Guidelines for ViT feature knowledge distillation]() Collection index: [llms.txt]() Data updated: 2025-10-27T06:16:31.191Z Authors: Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, Yu Li Published: 2022-09-06T11:52:46.000Z ## Abstract Knowledge Distillation (KD) for Convolutional Neural Network (CNN) is extensively studied as a way to boost the performance of a small model. Recently, Vision Transformer (ViT) has achieved great success on many computer vision tasks and KD for ViT is also desired. However, besides the output logit-based KD, other feature-based KD methods for CNNs cannot be directly applied to ViT due to the huge structure gap. In this paper, we explore the way of feature-based distillation for ViT. Based on the nature of feature maps in ViT, we design a series of controlled experiments and derive three practical guidelines for ViT's feature distillation. Some of our findings are even opposite to the practices in the CNN era. Based on the three guidelines, we propose our feature-based method ViTKD which brings consistent and considerable improvement to the student. On ImageNet-1k, we boost DeiT-Tiny from 74.42% to 76.06%, DeiT-Small from 80.55% to 81.95%, and DeiT-Base from 81.76% to 83.46%. Moreover, ViTKD and the logit-based KD method are complementary and can be applied together directly. This combination can further improve the performance of the student. Specifically, the student DeiT-Tiny, Small, and Base achieve 77.78%, 83.59%, and 85.41%, respectively. The code is available at https://github.com/yzd-v/cls\_KD. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Semantic Segmentation on ADE20K F (mIoU)](): Semantic segmentation; ADE20K F; mIoU: 43.2. - [Object Detection on COCO (mAP and Size APs)](): Object Detection; COCO; mAP: 45.8. - [Semantic Segmentation on Pascal VOC (default split)](): Semantic segmentation; Pascal VOC; mIoU: 77.2. - [Semantic Segmentation on ADE20K (mIoU)](): Semantic segmentation; ADE20K; mIoU: 45.5. - [Image Classification on ImageNet (val) (ACC)](): Image Classification; ImageNet (val); Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()