# MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

Source: [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](<https://www.sota2.com/research/paper/68fa6c2a1e209131cea32765>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:05:48.516Z

Authors: Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Published: 2021-10-05T02:14:38.000Z

## Abstract

Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks. Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/thunlp/MoEfication.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K (Accuracy and Latency)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-accuracy-and-latency.md>): Image Classification; ImageNet-1K; Accuracy: 99.

- [Computational Cost (GFLOPs/token) on Natural Language Generation for XSum](<https://www.sota2.com/research/sota/computational-cost-gflops-token-on-natural-language-generation-for-xsum.md>): Natural language generation; Xsum; GFLOPs/token: 8.27.

- [Language Modeling on WikiText-103 (Efficiency)](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-103-efficiency.md>): Language Modeling; WikiText-103; GFLOPs/token: 11.1.

- [GFLOPs per token on E2E for Natural Language Generation](<https://www.sota2.com/research/sota/gflops-per-token-on-e2e-for-natural-language-generation.md>): Natural language generation; E2E; GFLOPs per token: 7.39.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2110.01786>)

[PDF](<https://arxiv.org/pdf/2110.01786.pdf>)