# Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers Source: [Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers]() Collection index: [llms.txt]() Data updated: 2025-10-27T05:41:12.037Z Authors: Lucas Maisonnave, Karim Haroun, Tom Pegeot Published: 2025-08-22T11:43:39.000Z ## Abstract Transformer models rely on Multi-Head Self-Attention (MHSA) mechanisms, where each attention head contributes to the final representation. However, their computational complexity and high memory demands due to MHSA hinders their deployment at the edge. In this work, we analyze and exploit information redundancy in attention maps to accelerate model inference. By quantifying the information captured by each attention head using Shannon entropy, our analysis reveals that attention heads with lower entropy, i.e., exhibiting more deterministic behavior, tend to contribute less information, motivating targeted compression strategies. Relying on these insights, we propose Entropy Attention Maps (EAM), a model that freezes the weights of low-entropy attention maps and quantizes these values to low precision to avoid redundant re-computation. Empirical validation on ImageNet-1k shows that EAM achieves similar or higher accuracy at $\\leq$20\\% sparsity in attention maps and competitive performance beyond this level for the DeiT and Swin Transformer models. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()