# Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

Source: [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](<https://www.sota2.com/research/paper/68fa69931e209131cea22cad>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:03:46.301Z

Authors: Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, Anand Raghunathan

Published: 2021-03-16T20:04:09.000Z

## Abstract

Transformers have transformed the field of natural language processing. This performance is largely attributed to the use of stacked self-attention layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Hardware Sequence Processing Performance on Sequence length n=64 at 10 GBaud](<https://www.sota2.com/research/sota/hardware-sequence-processing-performance-on-sequence-length-n-64-at-10-gbaud.md>): Hardware Sequence Processing Performance; Sequence length n=64 at 10 GBaud; Latency (s): 0.00077.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2103.09301>)

[PDF](<https://arxiv.org/pdf/2103.09301.pdf>)
