# Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

Source: [Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?](<https://www.sota2.com/research/paper/6a0beebbdbc836b87d82c5b6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-20T05:00:05.687Z

Authors: Prateek Rajput, Yewei Song, Iyiola E. Olatunji, Jacques Klein, Tegawend\\'e F. Bissyand\\'e

Published: 2026-05-16T13:44:06.000Z

## Abstract

Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To investigate this, we induce personality in LLMs by fine-tuning them on the long-form essays, where each essay is associated with a target Big Five personality profile. We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire. Specifically, we ask: (i) does post-training (SFT, DPO, ORPO) stabilize questionnaire scores under prompt rephrasings, and (ii) can it induce target Big Five profiles from unguided essays? Our results demonstrate that fine-tuning consistently reduces variance in questionnaire responses across five models, directly mitigating the evaluation fragility reported in pre-trained models. However, this newfound stability reveals a more fundamental limitation: accuracy on the full five-dimensional profile remains near chance, even when single-trait scores improve. This indicates that unguided essays lack the cues needed for faithful personality expression. We therefore argue for scenario-grounded datasets or interactive elicitation that accumulates test-aligned evidence over time.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Personality Induction on Big-Five Personality Induction Without Questionnaires](<https://www.sota2.com/research/sota/personality-induction-on-big-five-personality-induction-without-questionnaires.md>): Personality Induction; Big-Five Personality Induction Without Questionnaires; Exact Match Accuracy (S1): 9.38.

- [Personality Induction on Big-Five Personality Induction With Questionnaires](<https://www.sota2.com/research/sota/personality-induction-on-big-five-personality-induction-with-questionnaires.md>): Personality Induction; Big-Five Personality Induction With Questionnaires; Accuracy (S1): 9.38.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.16996>)

[PDF](<https://arxiv.org/pdf/2605.16996.pdf>)