# IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation

Source: [IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation](<https://www.sota2.com/research/paper/68fa75131e209131cea658ab>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T07:20:50.585Z

Authors: Fan Lin, Shuyi Xie, Yong Dai, Wenlin Yao, Tianjiao Lang, Zishan Xu, Zhichao Hu, Xiao Xiao, Yuhong Liu, Yu Zhang

Published: 2024-09-27T16:29:12.000Z

## Abstract

As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according to model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Instruction Dataset Quality Evaluation on Public Instruction Tuning Datasets (test)](<https://www.sota2.com/research/sota/instruction-dataset-quality-evaluation-on-public-instruction-tuning-datasets-test.md>): Instruction Dataset Quality Evaluation; Public Instruction Tuning Datasets (test); Discrimination Index: 20.4.

- [Instruction Following Evaluation on WizardLM](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-wizardlm.md>): Instruction Following Evaluation; WizardLM; Score: not recorded.

- [Instruction Following Evaluation on Instruction Tuning with GPT-4](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-instruction-tuning-with-gpt-4.md>): Instruction Following Evaluation; Instruction Tuning with GPT-4; Score: not recorded.

- [Instruction Following Evaluation on SELF-INSTRUCT seed data](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-self-instruct-seed-data.md>): Instruction Following Evaluation; SELF-INSTRUCT seed data; Score: not recorded.

- [Instruction Following Evaluation on SELF-INSTRUCT](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-self-instruct.md>): Instruction Following Evaluation; Self-Instruct; Score: not recorded.

- [Instruction Following Evaluation on SELF-INSTRUCT Ours](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-self-instruct-ours.md>): Instruction Following Evaluation; SELF-INSTRUCT Ours; Score: not recorded.

- [Instruction Following Evaluation on Ours hard seed data](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-ours-hard-seed-data.md>): Instruction Following Evaluation; Ours hard seed data; Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.18892>)

[PDF](<https://arxiv.org/pdf/2409.18892.pdf>)

[Implementation](<https://github.com/DUTlf/IDGen.git>)
