# Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation

Source: [Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation](<https://www.sota2.com/research/paper/6a06a899dbc836b87d82a04e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-08-13T05:00:13.437Z

Authors: Kieran Carrigg, Sigur de Vries, Amirhossein Sadough, Marcel van Gerven

Published: 2026-05-13T19:08:55.000Z

## Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance on challenging vision tasks, but their deployment on edge devices is severely hindered by the computational complexity and global reduction bottleneck imposed by layer normalization. Recent methods attempt to bypass this by replacing normalization layers with hardware-friendly scalar approximations. However, these homogeneous replacements do not optimally fit to all layers' behaviour and rely on expensive model retraining. In this work, we propose a highly efficient, hardware-aware framework that utilizes genetic programming (GP) to evolve heterogeneous, layer-specific scalar functions directly from pre-trained weights. Coupled with a novel post-training re-alignment strategy, our approach eliminates the need to retrain models from scratch entirely. Our evolved expressions accurately approximate the target normalization behaviours, capturing $90\\%$ of the variance ($R^2$) compared to only $70.2\\%$ for homogeneous baselines, allowing our modified architecture to recover $84.32\\%$ Top-1 ImageNet-1K accuracy in only 20 epochs. By preserving this performance while eliminating the global reduction bottleneck, our approach achieves a strict reduction in both arithmetic complexity and off-chip memory traffic compared to standard LayerNorm, removing a primary barrier to the efficient deployment of ViTs on edge accelerators.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet 1K (val) (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-val-standard-accuracy.md>): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 84.25.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.14047>)

[PDF](<https://arxiv.org/pdf/2605.14047.pdf>)
