# Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks Source: [Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks]() Collection index: [llms.txt]() Data updated: 2025-10-26T04:45:08.569Z Authors: Uranik Berisha, Jens Mehnert, Alexandru Paul Condurache Published: 2025-05-21T11:55:25.000Z ## Abstract Vision Transformers have emerged as the state-of-the-art models in various Computer Vision tasks, but their high computational and resource demands pose significant challenges. While Mixture-of-Experts (MoE) can make these models more efficient, they often require costly retraining or even training from scratch. Recent developments aim to reduce these computational costs by leveraging pretrained networks. These have been shown to produce sparse activation patterns in the Multi-Layer Perceptrons (MLPs) of the encoder blocks, allowing for conditional activation of only relevant subnetworks for each sample. Building on this idea, we propose a new method to construct MoE variants from pretrained models. Our approach extracts expert subnetworks from the model's MLP layers post-training in two phases. First, we cluster output activations to identify distinct activation patterns. In the second phase, we use these clusters to extract the corresponding subnetworks responsible for producing them. On ImageNet-1k recognition tasks, we demonstrate that these extracted experts can perform surprisingly well out of the box and require only minimal fine-tuning to regain 98% of the original performance, all while reducing MACs and model size, by up to 36% and 32% respectively. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on ImageNet-1K (Accuracy and Latency)](): Image Classification; ImageNet-1K; Accuracy: 98. ## Original sources and implementation [Original paper]() [PDF]()