# SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

Source: [SKDBERT: Compressing BERT via Stochastic Knowledge Distillation](<https://www.sota2.com/research/paper/68fa6e071e209131cea3d811>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:48:10.793Z

Authors: Zixiang Ding, Guoqing Jiang, Shuai Zhang, Lin Guo, Wei Lin

Published: 2022-11-26T03:18:55.000Z

## Abstract

In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a pre-defined teacher ensemble, which consists of multiple teacher models with multi-level capacities, to transfer knowledge into student model in an one-to-one manner. Sampling distribution plays an important role in SKD. We heuristically present three types of sampling distributions to assign appropriate probabilities for multi-level teacher models. SKD has two advantages: 1) it can preserve the diversities of multi-level teacher models via stochastically sampling single teacher model in each iteration, and 2) it can also improve the efficacy of knowledge distillation via multi-level teacher models when large capacity gap exists between the teacher model and the student model. Experimental results on GLUE benchmark show that SKDBERT reduces the size of a BERT$\_{\\rm BASE}$ model by 40% while retaining 99.5% performances of language understanding and being 100% faster.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-100 (Raw Accuracy and GRR)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-raw-accuracy-and-grr.md>): Image Classification; CIFAR-100; Raw Accuracy: 92.37.

- [Image Classification on ImageNet (Raw Accuracy and GRR)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-raw-accuracy-and-grr.md>): Image Classification; ImageNet; Raw Accuracy: 78.4.

- [Rating Prediction on MovieLens (MSE, GRR)](<https://www.sota2.com/research/sota/rating-prediction-on-movielens-mse-grr.md>): Rating Prediction; MovieLens; MSE: 0.8411.

- [Image Classification on CIFAR-10 (Raw Accuracy and GRR)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-raw-accuracy-and-grr.md>): Image Classification; CIFAR-10; Raw Accuracy: 97.63.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2211.14466>)

[PDF](<https://arxiv.org/pdf/2211.14466.pdf>)