# On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

Source: [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](<https://www.sota2.com/research/paper/68fa6f921e209131cea46581>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T02:51:04.502Z

Authors: Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, Diyi Yang

Published: 2022-12-15T18:59:32.000Z

## Abstract

Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks. However, prior work has mainly focused on logical reasoning tasks (e.g. arithmetic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts. Concretely, we perform a controlled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks. We find that zero-shot CoT reasoning in sensitive domains significantly increases a model's likelihood to produce harmful or undesirable output, with trends holding across different prompt formats and model variants. Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following. Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Red-teaming on Web-Augmented LLM Red-Teaming Evaluation Set](<https://www.sota2.com/research/sota/red-teaming-on-web-augmented-llm-red-teaming-evaluation-set.md>): Red Teaming; Web-Augmented LLM Red-Teaming Evaluation Set; Detection Rate: 10.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2212.08061>)

[PDF](<https://arxiv.org/pdf/2212.08061.pdf>)

[Implementation](<https://github.com/SALT-NLP/chain-of-thought-bias>)
