# Exclusive Self Attention

Source: [Exclusive Self Attention](<https://www.sota2.com/research/paper/69b0f6f82d603e6de3e45c92>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-12T05:00:04.726Z

Authors: Shuangfei Zhai

Published: 2026-03-10T01:39:31.000Z

## Abstract

We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Zero-shot Evaluation on ARC-E, BoolQ, HSwag, LAMBADA, OBQA, PIQA, SocIQA, and WinoGr (test)](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-e-boolq-hswag-lambada-obqa-piqa-sociqa-and-winogr-test.md>): Zero-shot Evaluation; ARC-E, BoolQ, HSwag, LAMBADA, OBQA, PIQA, SocIQA, WinoGr (test); ARC-E Accuracy: 60.65.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.09078>)

[PDF](<https://arxiv.org/pdf/2603.09078.pdf>)
