# A Study on ReLU and Softmax in Transformer Source: [A Study on ReLU and Softmax in Transformer]() Collection index: [llms.txt]() Data updated: 2025-10-26T01:00:51.913Z Authors: Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, Jiang Bian Published: 2023-02-13T15:41:20.000Z ## Abstract The Transformer architecture consists of self-attention and feed-forward networks (FFNs) which can be viewed as key-value memories according to previous works. However, FFN and traditional memory utilize different activation functions (i.e., ReLU and Softmax respectively), which makes them not equivalent. In this paper, we first rebuild the connections between FFN and key-value memory by conducting extensive studies on ReLU and Softmax, and find they are equivalent when adding an additional layer normalization module on Softmax. In addition, ReLU outperforms Softmax on both FFN and key-value memory when the number of value slots is large. We analyze the reasons and then explore this good property of ReLU on the self-attention network where the original Softmax activation performs poorly on long input sequences. We then propose a full ReLU architecture named ReLUFormer which performs better than the baseline Transformer on long sequence tasks such as document translation. This paper sheds light on the following points: 1) Softmax and ReLU use different normalization methods over elements which lead to different variances of results, and ReLU is good at dealing with a large number of key-value slots; 2) FFN and key-value memory are equivalent, and thus the Transformer can be viewed as a memory network where FFNs and self-attention networks are both key-value memories. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Language Modeling on FineWeb 10B (1K sequence length, val)](): Language Modeling; FineWeb 10B 1K sequence length; Validation Loss: 3.2155. - [Language Modeling on FineWeb 10B (2K sequence length)](): Language Modeling; FineWeb 10B 2K sequence length; Validation Loss: 6.5573. - [Language Modeling on FineWeb 10B (4K sequence length, val)](): Language Modeling; FineWeb 10B 4K sequence length; Validation Loss: 8.9072. - [Language Modeling on FineWeb 10B (8K sequence length)](): Language Modeling; FineWeb 10B 8K sequence length; Validation Loss: 10.7266. ## Original sources and implementation [Original paper]() [PDF]()