# Nearest Neighbor Speculative Decoding for LLM Generation and Attribution Source: [Nearest Neighbor Speculative Decoding for LLM Generation and Attribution]() Collection index: [llms.txt]() Data updated: 2025-12-25T21:33:31.023Z Authors: Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen-tau Yih, Xi Victoria Lin Published: 2024-05-29T17:55:03.000Z ## Abstract Large language models (LLMs) often hallucinate and lack the ability to provide attribution for their generations. Semi-parametric LMs, such as kNN-LM, approach these limitations by refining the output of an LM for a given prompt using its nearest neighbor matches in a non-parametric data store. However, these models often exhibit slow inference speeds and produce non-fluent texts. In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a novel semi-parametric language modeling approach that is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources. NEST performs token-level retrieval at each inference step to compute a semi-parametric mixture distribution and identify promising span continuations in a corpus. It then uses an approximate speculative decoding procedure that accepts a prefix of the retrieved span or generates a new token. NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks, surpassing the conventional kNN-LM method and performing competitively with in-context retrieval augmentation. In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B. Code will be released at https://github.com/facebookresearch/NEST/tree/main. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Long-form Question Answering on Biography](): Long-form Question Answering; Biography; VeriScore F1: 41.8. - [Long-form Question Answering on LongFact](): Long-form Question Answering; LongFact; VeriScore F1: 63.2. - [Long-form Question Answering on Fava](): Long-form Question Answering; Fava; VeriScore F1: 50.3. - [Long-form Question Answering on AlpacaFact](): Long-form Question Answering; AlpacaFact; VeriScore F1: 58.1. ## Original sources and implementation [Original paper]() [PDF]()