# Transformer Meets Twicing: Harnessing Unattended Residual Information

Source: [Transformer Meets Twicing: Harnessing Unattended Residual Information](<https://www.sota2.com/research/paper/68fa7a651e209131cea82e87>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:59:41.376Z

Authors: Laziz Abdullaev, Tan M. Nguyen

Published: 2025-03-02T01:56:35.000Z

## Abstract

Transformer-based deep learning models have achieved state-of-the-art performance across numerous language and vision tasks. While the self-attention mechanism, a core component of transformers, has proven capable of handling complex data patterns, it has been observed that the representational capacity of the attention matrix degrades significantly across transformer layers, thereby hurting its overall performance. In this work, we leverage the connection between self-attention computations and low-pass non-local means (NLM) smoothing filters and propose the Twicing Attention, a novel attention mechanism that uses kernel twicing procedure in nonparametric regression to alleviate the low-pass behavior of associated NLM smoothing with compelling theoretical guarantees and enhanced adversarial robustness. This approach enables the extraction and reuse of meaningful information retained in the residuals following the imperfect smoothing operation at each layer. Our proposed method offers two key advantages over standard self-attention: 1) a provably slower decay of representational capacity and 2) improved robustness and accuracy across various data modalities and tasks. We empirically demonstrate the performance gains of our model over baseline transformers on multiple tasks and benchmarks, including image classification and language modeling, on both clean and corrupted data.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on WikiText-103 (val)](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-103-val.md>): Language Modeling; WikiText-103 (val); PPL: 70.3.

- [Language Modeling on WikiText-103 (test)](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-103-test.md>): Language Modeling; WikiText-103 (test); Perplexity: 69.6.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2503.00687>)

[PDF](<https://arxiv.org/pdf/2503.00687.pdf>)
