# Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

Source: [Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment](<https://www.sota2.com/research/paper/68fa72f41e209131cea59867>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:38:38.445Z

Authors: Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen

Published: 2023-11-07T15:36:40.000Z

## Abstract

Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which cannot fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quality signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing a number of competitive baselines.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-turn Response Generation on MT-Bench (test)](<https://www.sota2.com/research/sota/multi-turn-response-generation-on-mt-bench-test.md>): Multi-turn response generation; MT-Bench (test); Turn 1 Score: 5.11.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2311.04072>)

[PDF](<https://arxiv.org/pdf/2311.04072.pdf>)