# QUILL: Quotation Generation Enhancement of Large Language Models

Source: [QUILL: Quotation Generation Enhancement of Large Language Models](<https://www.sota2.com/research/paper/68fa77101e209131cea70e56>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:47:55.951Z

Authors: Jin Xiao, Bowei Zhang, Qianyu He, Jiaqing Liang, Feng Wei, Jinglei Chen, Zujie Liang, Deqing Yang, Yanghua Xiao

Published: 2024-11-06T05:24:09.000Z

## Abstract

While Large language models (LLMs) have become excellent writing assistants, they still struggle with quotation generation. This is because they either hallucinate when providing factual quotations or fail to provide quotes that exceed human expectations. To bridge the gap, we systematically study how to evaluate and improve LLMs' performance in quotation generation tasks. We first establish a holistic and automatic evaluation system for quotation generation task, which consists of five criteria each with corresponding automatic metric. To improve the LLMs' quotation generation abilities, we construct a bilingual knowledge base that is broad in scope and rich in dimensions, containing up to 32,022 quotes. Moreover, guided by our critiria, we further design a quotation-specific metric to rerank the retrieved quotations from the knowledge base. Extensive experiments show that our metrics strongly correlate with human preferences. Existing LLMs struggle to generate desired quotes, but our quotation knowledge base and reranking metric help narrow this gap. Our dataset and code are publicly available at https://github.com/GraceXiaoo/QUILL.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Quotation Recommendation on QuoteR](<https://www.sota2.com/research/sota/quotation-recommendation-on-quoter.md>): Quotation Recommendation; QuoteR; Novelty: 3.42.

- [Quotation Recommendation on QUILL](<https://www.sota2.com/research/sota/quotation-recommendation-on-quill.md>): Quotation Recommendation; QUILL; Novelty: 3.32.

- [Quotation Recommendation on NOVELQR-BENCH](<https://www.sota2.com/research/sota/quotation-recommendation-on-novelqr-bench.md>): Quotation Recommendation; NOVELQR-BENCH; Novelty Score: 3.08.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.03675>)

[PDF](<https://arxiv.org/pdf/2411.03675.pdf>)
