# SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy

Source: [SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy](<https://www.sota2.com/research/paper/69d33e107d0172140ec73704>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-07T05:00:04.607Z

Authors: Joy Bhalla, Kristina Gligori\\'c

Published: 2026-04-02T18:00:14.000Z

## Abstract

Large language models exhibit sycophancy: the tendency to shift outputs toward user-expressed stances, regardless of correctness or consistency. While prior work has studied this issue and its impacts, rigorous computational linguistic metrics are needed to identify when models are being sycophantic. Here, we introduce SWAY, an unsupervised computational linguistic measure of sycophancy. We develop a counterfactual prompting mechanism to identify how much a model's agreement shifts under positive versus negative linguistic pressure, isolating framing effects from content. Applying this metric to benchmark 6 models, we find that sycophancy increases with epistemic commitment. Leveraging our metric, we introduce a counterfactual mitigation strategy teaching models to consider what the answer would be if opposite assumptions were suggested. While baseline mitigation instructing to be explicitly anti-sycophantic yields moderate reductions, and can backfire, our counterfactual CoT mitigation drives sycophancy to near zero across models, commitment levels, and clause types, while not suppressing responsiveness to genuine evidence. Overall, we contribute a metric for benchmarking sycophancy and a mitigation informed by it.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Sycophancy on LFQA](<https://www.sota2.com/research/sota/sycophancy-on-lfqa.md>): Sycophancy; LFQA; Sycophancy (PD, L): not recorded.

- [Sycophancy Evaluation on AITA](<https://www.sota2.com/research/sota/sycophancy-evaluation-on-aita.md>): Sycophancy Evaluation; AITA; Sycophancy Score (S) PD-L: not recorded.

- [Sycophancy Evaluation on DebateQA](<https://www.sota2.com/research/sota/sycophancy-evaluation-on-debateqa.md>): Sycophancy Evaluation; DebateQA; S (PD, L): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.02423>)

[PDF](<https://arxiv.org/pdf/2604.02423.pdf>)
