# Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension

Source: [Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension](<https://www.sota2.com/research/paper/69e5b2d97d0172140ec7dde4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-21T05:00:04.784Z

Authors: Dongxin Guo, Jikun Wu, Siu Ming Yiu

Published: 2026-04-17T07:15:53.000Z

## Abstract

Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57\\times$ energy efficiency on neuromorphic hardware, yet no theoretical framework guides their design. This paper establishes the first comprehensive expressivity theory for spiking self-attention. We prove that spiking attention with Leaky Integrate-and-Fire neurons is a universal approximator of continuous permutation-equivariant functions, providing explicit spike circuit constructions including a novel lateral inhibition network for softmax normalization with proven $O(1/\\sqrt{T})$ convergence. We derive tight spike-count lower bounds via rate-distortion theory: $\\varepsilon$-approximation requires $\\Omega(L\_f^2 nd/\\varepsilon^2)$ spikes, with rigorous information-theoretic derivation. Our key insight is input-dependent bounds using measured effective dimensions ($d\_{\\text{eff}}=47$--$89$ for CIFAR/ImageNet), explaining why $T=4$ timesteps suffice despite worst-case $T \\geq 10{,}000$ predictions. We provide concrete design rules with calibrated constants ($C=2.3$, 95\\% CI: $\[1.9, 2.7\]$). Experiments on Spikformer, QKFormer, and SpikingResformer across vision and language benchmarks validate predictions with $R^2=0.97$ ($p\<0.001$). Our framework provides the first principled foundation for neuromorphic transformer design.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K 1.0 (test) (Accuracy and Energy Efficiency)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-test-accuracy-and-energy-efficiency.md>): Image Classification; ImageNet-1k 1.0 (test); Accuracy: 84.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.15769>)

[PDF](<https://arxiv.org/pdf/2604.15769.pdf>)