# Learn To be Efficient: Build Structured Sparsity in Large Language Models Source: [Learn To be Efficient: Build Structured Sparsity in Large Language Models]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:40:09.295Z Authors: Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z. Morley Mao, Beidi Chen, Fan Lai, Atul Prakash Published: 2024-02-09T01:18:16.000Z ## Abstract Large Language Models (LLMs) have achieved remarkable success with their billion-level parameters, yet they incur high inference overheads. The emergence of activation sparsity in LLMs provides a natural approach to reduce this cost by involving only parts of the parameters for inference. However, existing methods only focus on utilizing this naturally formed activation sparsity in a post-training setting, overlooking the potential for further amplifying this inherent sparsity. In this paper, we hypothesize that LLMs can learn to be efficient by achieving more structured activation sparsity. To achieve this, we introduce a novel training algorithm, Learn-To-be-Efficient (LTE), designed to train efficiency-aware LLMs to learn to activate fewer neurons and achieve a better trade-off between sparsity and performance. Furthermore, unlike SOTA MoEfication methods, which mainly focus on ReLU-based models, LTE can also be applied to LLMs like LLaMA using non-ReLU activations. Extensive evaluation on language understanding, language generation, and instruction tuning tasks show that LTE consistently outperforms SOTA baselines. Along with our hardware-aware custom kernel implementation, LTE reduces LLaMA2-7B inference latency by 25% at 50% sparsity. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Language Modeling on WikiText-103 (fine-tuned)](): Language Modeling; WikiText103 fine-tuned; Perplexity (PPL): 5.95. - [Computational Cost (GFLOPs/token) on Natural Language Generation for XSum](): Natural language generation; Xsum; GFLOPs/token: 5.38. - [Language Modeling on WikiText-103 (Efficiency)](): Language Modeling; WikiText-103; GFLOPs/token: 6.59. - [GFLOPs per token on E2E for Natural Language Generation](): Natural language generation; E2E; GFLOPs per token: 4.65. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()