# Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data

Source: [Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Data](<https://www.sota2.com/research/paper/68fa77991e209131cea7445c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-18T05:00:11.457Z

Authors: Haoyan Yang, Khiem Le, Ting Hua, Shangqian Gao, Binfeng Xu, Zheng Tang, Jie Xu, Nitesh V. Chawla, Hongxia Jin, Vijay Srinivasan

Published: 2025-02-08T01:20:09.000Z

## Abstract

Although LLMs have achieved significant success, their reliance on large volumes of human-annotated data has limited their potential for further scaling. In this situation, utilizing self-generated synthetic data has become crucial for fine-tuning LLMs without extensive human annotation. However, current methods often fail to ensure consistent improvements across iterations, with performance stagnating after only minimal updates. To overcome these challenges, we introduce Dynamic Noise Preference Optimization (DNPO), which combines dynamic sample labeling for constructing preference pairs with controlled, trainable noise injection during preference optimization. Our approach effectively prevents stagnation and enables continuous improvement. In experiments with Llama-3.2-3B and Zephyr-7B, DNPO consistently outperforms existing methods across multiple benchmarks. Additionally, with Zephyr-7B, DNPO shows a significant improvement in model-generated data quality, with a 29.4% win-loss rate gap compared to the baseline in GPT-4 evaluations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Large Language Model Evaluation on ARC, TruthfulQA, Winogrande, GSM8K, HellaSwag, and MMLU](<https://www.sota2.com/research/sota/large-language-model-evaluation-on-arc-truthfulqa-winogrande-gsm8k-hellaswag-and-mmlu.md>): Large Language Model Evaluation; ARC, TruthfulQA, Winogrande, GSM8K, HellaSwag, MMLU; ARC Accuracy: 73.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2502.05400>)

[PDF](<https://arxiv.org/pdf/2502.05400.pdf>)