# Parameter-Efficient Neuroevolution for Diverse LLM Generation: Quality-Diversity Optimization via Prompt Embedding Evolution

Source: [Parameter-Efficient Neuroevolution for Diverse LLM Generation: Quality-Diversity Optimization via Prompt Embedding Evolution](<https://www.sota2.com/research/paper/6a02b449dbc836b87d824965>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-24T05:00:04.550Z

Authors: Dongxin Guo, Jikun Wu, Siu Ming Yiu

Published: 2026-05-10T22:00:15.000Z

## Abstract

Large Language Models exhibit mode collapse, producing homogeneous outputs that fail to explore valid solution spaces. We present QD-LLM, a framework for parameter-efficient neuroevolution that evolves prompt embeddings, compact neural interfaces (~32K parameters) that steer generation in frozen LLMs (70B+ parameters), within a Quality-Diversity (QD) optimization framework. Our contributions: (1) evolved prompt embeddings via gradient-free optimization enabling behavioral steering without model fine-tuning; (2) hybrid behavior characterization combining semantic and explicit features with formal coverage bounds (Theorem 1) under validated near-independence (NMI $= 0.08 \\pm 0.02$); (3) co-evolutionary variation operators including targeted behavioral mutation via finite-difference gradient estimation. On HumanEval (164 problems), MBPP, and creative writing benchmarks, QD-LLM achieves 46.4% higher coverage and 41.4% higher QD-Score than QDAIF ($p\<0.001$, 30 runs, Vargha-Delaney $A=0.94$). We demonstrate downstream utility: diverse archives improve test generation (34% more edge cases) and fine-tuning data quality (8.3% accuracy gain). We validate across open-source LLMs (Llama-3-70B, Mistral-Large) with full embedding access, establishing prompt embedding evolution as an effective paradigm bridging neuroevolution and modern LLMs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Code Generation on HumanEval Llama-3-70B (test)](<https://www.sota2.com/research/sota/code-generation-on-humaneval-llama-3-70b-test.md>): Code Generation; HumanEval Llama-3-70B (test); QD-Score: 26.3.

- [Creative Writing on ROCStories + WritingPrompts Llama-3-70B (test)](<https://www.sota2.com/research/sota/creative-writing-on-rocstories-plus-writingprompts-llama-3-70b-test.md>): Creative Writing; ROCStories + WritingPrompts Llama-3-70B (test); QD-Score: 63.3.

- [Creative Writing on ROCStories + WritingPrompts](<https://www.sota2.com/research/sota/creative-writing-on-rocstories-plus-writingprompts.md>): Creative Writing; ROCStories + WritingPrompts; QD-Score: 63.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.09781>)

[PDF](<https://arxiv.org/pdf/2605.09781.pdf>)