# LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

Source: [LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel](<https://www.sota2.com/research/paper/69e9a7b655a3c32af89f7225>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-24T05:00:04.605Z

Authors: Zhe Feng, Sen Lian, Changwei Wang, Muyang Zhang, Tianlong Tan, Rongtao Xu, Weiliang Meng, Xiaopeng Zhang

Published: 2026-04-22T09:04:54.000Z

## Abstract

The quadratic complexity of softmax attention presents a major obstacle for scaling Transformers to high-resolution vision tasks. Existing linear attention variants often replace the softmax with Gaussian kernels to reduce complexity, but such approximations lack theoretical grounding and tend to oversuppress mid-range token interactions. We propose LaplacianFormer, a Transformer variant that employs a Laplacian kernel as a principled alternative to softmax, motivated by empirical observations and theoretical analysis. To address expressiveness degradation under low-rank approximations, we introduce a provably injective feature map that retains fine-grained token information. For efficient computation, we adopt a Nystr\\"om approximation of the kernel matrix and solve the resulting system using Newton--Schulz iteration, avoiding costly matrix inversion and SVD. We further develop custom CUDA implementations for both the kernel and solver, enabling high-throughput forward and backward passes suitable for edge deployment. Experiments on ImageNet show that LaplacianFormer achieves strong performance-efficiency trade-offs while improving attention expressiveness.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet 1K (val) (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-val-standard-accuracy.md>): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 85.8.

- [Instance Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017-val.md>): Instance Segmentation; COCO 2017 (val); APm: 0.438.

- [Object Detection on COCO 2017 (val)](<https://www.sota2.com/research/sota/object-detection-on-coco-2017-val.md>): Object Detection; COCO 2017 (val); AP: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.20368>)

[PDF](<https://arxiv.org/pdf/2604.20368.pdf>)