# SMYRF: Efficient Attention using Asymmetric Clustering

Source: [SMYRF: Efficient Attention using Asymmetric Clustering](<https://www.sota2.com/research/paper/68fa687d1e209131cea1c44b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:38:00.142Z

Authors: Giannis Daras, Nikita Kitaev, Augustus Odena, Alexandros G. Dimakis

Published: 2020-10-11T18:49:17.000Z

## Abstract

We propose a novel type of balanced clustering algorithm to approximate attention. Attention complexity is reduced from $O(N^2)$ to $O(N \\log N)$, where $N$ is the sequence length. Our algorithm, SMYRF, uses Locality Sensitive Hashing (LSH) in a novel way by defining new Asymmetric transformations and an adaptive scheme that produces balanced clusters. The biggest advantage of SMYRF is that it can be used as a drop-in replacement for dense attention layers without any retraining. On the contrary, prior fast attention methods impose constraints (e.g. queries and keys share the same vector representations) and require re-training from scratch. We apply our method to pre-trained state-of-the-art Natural Language Processing and Computer Vision models and we report significant memory and speed benefits. Notably, SMYRF-BERT outperforms (slightly) BERT on GLUE, while using $50\\%$ less memory. We also show that SMYRF can be used interchangeably with dense attention before and after training. Finally, we use SMYRF to train GANs with attention in high resolutions. Using a single TPU, we were able to scale attention to 128x128=16k and 256x256=65k tokens on BigGAN on CelebA-HQ.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (val) (Top-1 Accuracy and Error)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-top-1-accuracy-and-error.md>): Image Classification; ImageNet (val); Top-1 Accuracy: 79.8.

- [Natural Language Understanding on GLUE (dev)](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue-dev.md>): Natural Language Understanding; GLUE (dev); SST-2 (Acc): 93.2.

- [Natural Language Understanding on GLUE](<https://www.sota2.com/research/sota/natural-language-understanding-on-glue.md>): Natural Language Understanding; GLUE; SST-2: 91.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.05315>)

[PDF](<https://arxiv.org/pdf/2010.05315.pdf>)