# Decoupled Mixture-of-Experts for Parametric Knowledge Injection

Source: [Decoupled Mixture-of-Experts for Parametric Knowledge Injection](<https://www.sota2.com/research/paper/6a2f86e7d33dfb23e783be5c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-16T05:00:05.515Z

Authors: Baoqing Yue, Weihang Su, Qingyao Ai, Yichen Tang, Changyue Wang, Jiacheng Kang, Jingtao Zhan, Yiqun Liu

Published: 2026-06-12T08:21:28.000Z

## Abstract

Knowledge injection aims to equip large language models (LLMs) with external, domain-specific, or time-sensitive knowledge. Existing approaches typically face a trade-off between flexibility and integration: retrieval-augmented generation keeps knowledge outside the model but only provides prompt-level augmentation, whereas post-training based methods encode new knowledge into shared parameters but may introduce catastrophic forgetting, knowledge conflict, and costly updates. In this paper, we propose Decoupled Mixture-of-Experts (DMoE), a modular architecture for parametric knowledge injection that decouples both experts and the router from the base model. DMoE converts external knowledge corpora into independently updatable expert modules and uses a lightweight uncertainty-aware router to activate relevant experts only when the base model lacks sufficient knowledge during generation. To support efficient auto-regressive inference, DMoE attaches experts only to the final-layer feed-forward network, preserving KV-cache reuse while enabling parameter-level knowledge augmentation. Experiments on knowledge-intensive benchmarks show that DMoE consistently improves answer quality over retrieval and adapter-based baselines.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Question Answering on CWQ (EM, F1)](<https://www.sota2.com/research/sota/question-answering-on-cwq-em-f1.md>): Question Answering; CWQ; Exact Match (EM): 24.67.

- [Question Answering on CWQ, HotpotQA, Quasar-T, and StrategyQA Average (test)](<https://www.sota2.com/research/sota/question-answering-on-cwq-hotpotqa-quasar-t-and-strategyqa-average-test.md>): Question Answering; CWQ, HotpotQA, Quasar-T, and StrategyQA Average (test); Inference Time (s): 2.6656.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.14243>)

[PDF](<https://arxiv.org/pdf/2606.14243.pdf>)