# Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

Source: [Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models](<https://www.sota2.com/research/paper/6a34cd04206542251d7c5e84>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-20T05:00:05.764Z

Authors: Amogh Sheth, Biruk Assefa, Yi Wen Huang, Andrew Lin, Yuhao Ge

Published: 2026-04-27T01:44:10.000Z

## Abstract

Large language models (LLMs) excel at multi-step reasoning but incur substantial inference cost. We introduce Causal Attribution Pruning (CAP), a training-free method that identifies critical attention heads by measuring their causal impact on reasoning tasks and uses these head-level scores to guide fine-grained weight pruning. For each attention head, CAP estimates the expected performance degradation when the head is masked during forward passes on a small calibration set of reasoning problems. These causal scores are then converted into weight-level importance values for the corresponding projection matrices. Unlike magnitude-only or activation-based criteria, CAP's interventional measurement directly captures each head's functional contribution, yielding relative accuracy gains of up to 61% over Wanda on ARC-Challenge at 20% sparsity. We evaluate CAP on GSM8K, StrategyQA, and ARC-Challenge using Llama-3-8B-Instruct and Mistral-7B-Instruct at 10%, 20%, and 50% sparsity. At moderate sparsity (10-20%), CAP improves over Wanda in most model-benchmark configurations. with especially large gains on ARC-Challenge for Llama-3. Our results suggest that attention-head-level causal attribution can better preserve reasoning performance on downstream benchmarks than correlational pruning criteria at equivalent sparsity, while remaining limited by coarse MLP attribution at 50% sparsity.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Commonsense Reasoning on StrategyQA (accuracy)](<https://www.sota2.com/research/sota/commonsense-reasoning-on-strategyqa-accuracy.md>): Commonsense Reasoning; StrategyQA; Accuracy (StrategyQA): 65.5.

- [Language Modeling on WikiText-2](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-2.md>): Language Modeling; WikiText-2; Perplexity (PPL): 5.85.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.19350>)

[PDF](<https://arxiv.org/pdf/2606.19350.pdf>)
