# Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation

Source: [Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation](<https://www.sota2.com/research/paper/68fa760a1e209131cea6b0d0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:46:34.345Z

Authors: Yifang Chen, David Zhu, Simon Du, Kevin Jamieson, Yang Liu

Published: 2024-10-27T07:38:39.000Z

## Abstract

Recent advances in large language model (LLM) training have highlighted the need for diverse, high-quality instruction data. Recently, many works are exploring synthetic data generation using LLMs. However, they primarily focus on prompt engineering with standard supervised instruction-finetuned models, which contains a fundamental limitation: these models are optimized for general question-answering/problem-solving rather than data generation. We propose a paradigm shift named \\textbf{NOMAD} by investigating how to specifically train models for data generation, demonstrating that this task differs significantly from training a classical LM. We identify two key factors: no-prompt-masked training and proper training set size selection. Our method, NOMAD, shows substantial improvements over baselines, achieving \>4\\% gains in TriviaQA and \>2\\% in GSM8K with limited training data. Finally, we offer new insights by interpreting synthetic data through the lenses of "relevance" and "novelty".

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2410.20362>)

[PDF](<https://arxiv.org/pdf/2410.20362.pdf>)
