# Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval

Source: [Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval](<https://www.sota2.com/research/paper/68fa73aa1e209131cea5dc63>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:54:13.197Z

Authors: Zizhong Li, Haopeng Zhang, Jiawei Zhang

Published: 2024-06-18T00:41:41.000Z

## Abstract

Recent research has explored distilling knowledge from large language models (LLMs) to optimize retriever models, especially within the retrieval-augmented generation (RAG) framework. However, most existing training methods rely on extracting supervision signals from LLMs' weights or their output probabilities, which is not only resource-intensive but also incompatible with black-box LLMs. In this paper, we introduce \\textit{Intermediate Distillation}, a data-efficient knowledge distillation training scheme that treats LLMs as black boxes and distills their knowledge via an innovative LLM-ranker-retriever pipeline, solely using LLMs' ranking generation as the supervision signal. Extensive experiments demonstrate that our proposed method can significantly improve the performance of retriever models with only 1,000 training instances. Moreover, our distilled retriever model significantly boosts performance in question-answering tasks within the RAG framework, demonstrating the potential of LLMs to economically and effectively train smaller models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2406.12169>)

[PDF](<https://arxiv.org/pdf/2406.12169.pdf>)
