# Embedding Recycling for Language Models Source: [Embedding Recycling for Language Models]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:56:59.119Z Authors: Jon Saad-Falcon, Amanpreet Singh, Luca Soldaini, Mike D'Arcy, Arman Cohan, Doug Downey Published: 2022-07-11T16:36:14.000Z ## Abstract Real-world applications of neural language models often involve running many different models over the same corpus. The high computational cost of these runs has led to interest in techniques that can reuse the contextualized embeddings produced in previous runs to speed training and inference of future ones. We refer to this approach as embedding recycling (ER). While multiple ER techniques have been proposed, their practical effectiveness is still unknown because existing evaluations consider very few models and do not adequately account for overhead costs. We perform an extensive evaluation of ER across eight different models (17 to 900 million parameters) and fourteen tasks in English. We show how a simple ER technique that caches activations from an intermediate layer of a pretrained model, and learns task-specific adapters on the later layers, is broadly effective. For the best-performing baseline in our experiments (DeBERTa-v2 XL), adding a precomputed cache results in a \>90% speedup during training and 87-91% speedup for inference, with negligible impact on accuracy. Our analysis reveals important areas of future work. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Knowledge-intensive NLP tasks on FEVER, NQ, WoW, and ZSRE](): Knowledge-intensive NLP tasks; FEVER, NQ, WoW, and ZSRE; Average Performance: 35.54. - [Knowledge-Intensive Language Tasks on KILT (test)](): Knowledge-Intensive Language Tasks; KILT (test); WoW F1 Score: 17.3. ## Original sources and implementation [Original paper]() [PDF]()