# SPECTRE: An FFT-Based Efficient Drop-In Replacement to Self-Attention for Long Contexts Source: [SPECTRE: An FFT-Based Efficient Drop-In Replacement to Self-Attention for Long Contexts]() Collection index: [llms.txt]() Data updated: 2025-10-26T04:41:15.331Z Authors: Jacob Fein-Ashley, Neelesh Gupta, Rajgopal Kannan, Viktor Prasanna Published: 2025-02-25T17:43:43.000Z ## Abstract Long-context transformers face significant efficiency challenges due to the quadratic cost of self-attention. However, many modern applications-from multi-turn dialogue to high-resolution vision-require contexts spanning tens of thousands of tokens. We introduce SPECTRE, a method that replaces each attention head with a fast real FFT, a content-adaptive spectral gate, and an inverse FFT, reducing per-layer complexity from $\\mathcal{O}(L^{2})$ to $O(L\\log L)$ while preserving the surrounding architecture. We extend this efficiency to autoregressive generation through our Prefix-FFT cache and enhance local feature representation with an optional wavelet module that adds negligible computational overhead. Our experiments demonstrate that SPECTRE operates up to 7$\\times$ faster than FlashAttention-2 on 128k-token contexts while matching or exceeding baseline performance on PG-19 language modeling and ImageNet-1k classification tasks. SPECTRE achieves these improvements by adding fewer than 6\\% parameters to the base model, making hundred-kilotoken context processing feasible on commodity GPUs without specialized hardware. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()