# BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

Source: [BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity](<https://www.sota2.com/research/paper/68fa7a451e209131cea8213e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:16:12.275Z

Authors: Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Yuxuan Li, Zhiyuan Liu, Maosong Sun

Published: 2025-07-11T17:28:56.000Z

## Abstract

To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing attention. However, the non-differentiable and inflexible routing of vanilla MoE hurts model performance. Moreover, while each token activates only a few parameters, these sparsely-activated architectures exhibit low chunk-level sparsity, indicating that the union of multiple consecutive tokens activates a large ratio of parameters. Such a sparsity pattern is unfriendly for acceleration under low-resource conditions (e.g., end-side devices) and incompatible with mainstream acceleration techniques (e.g., speculative decoding). To address these challenges, we introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques. Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing. Next, to promote both token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly. Finally, we implement efficient acceleration kernels, combining activation sparsity and speculative decoding for the first time. The experimental results demonstrate the superior performance of BlockFFN over other MoE baselines, achieving over 80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67$\\times$ speedup on real end-side devices than dense models. All codes and checkpoints are available publicly (https://github.com/thunlp/BlockFFN).

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Zero-shot Language Understanding on Commonsense Reasoning Suite (PIQA, SIQA, HellaSwag, ARC, WinoGrande, LAMBADA)](<https://www.sota2.com/research/sota/zero-shot-language-understanding-on-commonsense-reasoning-suite-piqa-siqa-hellaswag-arc-winogrande-lambada.md>): Zero-shot Language Understanding; Commonsense Reasoning and Language Modeling Suite (PIQA, SIQA, HellaSwag, ARC-E, ARC-C, WinoGrande, LAMBADA) zero-shot; PIQA Accuracy: 70.02.

- [Zero-shot Commonsense Reasoning Suite (PIQA, SIQA, HellaSwag, ARC-E, ARC-C, WinoGrande, LAMBADA)](<https://www.sota2.com/research/sota/zero-shot-commonsense-reasoning-suite-piqa-siqa-hellaswag-arc-e-arc-c-winogrande-lambada.md>): Commonsense Reasoning; Commonsense Reasoning Suite (PIQA, SIQA, HellaSwag, ARC-E, ARC-C, WinoGrande, LAMBADA) zero-shot; PIQA Accuracy: 66.1.

- [Commonsense Reasoning on Commonsense Reasoning Suite (Zero-shot)](<https://www.sota2.com/research/sota/commonsense-reasoning-on-commonsense-reasoning-suite-zero-shot.md>): Commonsense Reasoning; Commonsense Reasoning Suite Zero-shot; PIQA Accuracy: 63.38.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.08771>)

[PDF](<https://arxiv.org/pdf/2507.08771.pdf>)