# Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers

Source: [Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers](<https://www.sota2.com/research/paper/68fa76d61e209131cea6f6e6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:13.306Z

Authors: Mark Horton, Tergel Molom-Ochir, Peter Liu, Bhavna Gopal, Chiyue Wei, Cong Guo, Brady Taylor, Deliang Fan, Shan X. Wang, Hai Li, Yiran Chen

Published: 2025-02-03T19:24:01.000Z

## Abstract

Pre-trained transformer models with extended context windows are notoriously expensive to run at scale, often limiting real-world deployment due to their high computational and memory requirements. In this paper, we introduce Hamming Attention Distillation (HAD), a novel framework that binarizes keys and queries in the attention mechanism to achieve significant efficiency gains. By converting keys and queries into {-1, +1} vectors and replacing dot-product operations with efficient Hamming distance computations, our method drastically reduces computational overhead. Additionally, we incorporate attention matrix sparsification to prune low-impact activations, which further reduces the cost of processing long-context sequences. \\par Despite these aggressive compression strategies, our distilled approach preserves a high degree of representational power, leading to substantially improved accuracy compared to prior transformer binarization methods. We evaluate HAD on a range of tasks and models, including the GLUE benchmark, ImageNet, and QuALITY, demonstrating state-of-the-art performance among binarized Transformers while drastically reducing the computational costs of long-context inference. \\par We implement HAD in custom hardware simulations, demonstrating superior performance characteristics compared to a custom hardware implementation of standard attention. HAD achieves just $\\mathbf{1.78}\\%$ performance losses on GLUE compared to $9.08\\%$ in state-of-the-art binarization work, and $\\mathbf{2.5}\\%$ performance losses on ImageNet compared to $12.14\\%$, all while targeting custom hardware with a $\\mathbf{79}\\%$ area reduction and $\\mathbf{87}\\%$ power reduction compared to its standard attention counterpart.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2502.01770>)

[PDF](<https://arxiv.org/pdf/2502.01770.pdf>)