# Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks

Source: [Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks](<https://www.sota2.com/research/paper/68fa79ee1e209131cea800e4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:18.887Z

Authors: Joyeeta Datta, Niclas Doll, Qusai Ramadan, Zeyd Boukhers

Published: 2025-07-10T10:54:05.000Z

## Abstract

Large Language Models (LLMs) have demonstrated outstanding performance across a range of NLP tasks, however, their computational demands hinder their deployment in real-world, resource-constrained environments. This work investigates the extent to which LLMs can be compressed using Knowledge Distillation (KD) while maintaining strong performance on Question Answering (QA) tasks. We evaluate student models distilled from the Pythia and Qwen2.5 families on two QA benchmarks, SQuAD and MLQA, under zero-shot and one-shot prompting conditions. Results show that student models retain over 90% of their teacher models' performance while reducing parameter counts by up to 57.1%. Furthermore, one-shot prompting yields additional performance gains over zero-shot setups for both model families. These findings underscore the trade-off between model efficiency and task performance, demonstrating that KD, combined with minimal prompting, can yield compact yet capable QA systems suitable for resource-constrained applications.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.07630>)

[PDF](<https://arxiv.org/pdf/2507.07630.pdf>)
