# DEMix Layers: Disentangling Domains for Modular Language Modeling

Source: [DEMix Layers: Disentangling Domains for Modular Language Modeling](<https://www.sota2.com/research/paper/68fa6a981e209131cea29372>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:29:26.390Z

Authors: Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer

Published: 2021-08-11T05:15:33.000Z

## Abstract

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer is a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation with little overhead. We show that mixing experts during inference, using a parameter-free weighted ensemble, allows the model to better generalize to heterogeneous or unseen domains. We also show that experts can be added to iteratively incorporate new domains without forgetting older ones, and that experts can be removed to restrict access to unwanted domains, without additional training. Overall, these results demonstrate benefits of explicitly conditioning on textual domains during language modeling.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text Classification on Restaurant, AI, ACL, AGNews Continual Learning Sequence (test)](<https://www.sota2.com/research/sota/text-classification-on-restaurant-ai-acl-agnews-continual-learning-sequence-test.md>): Text Classification; Restaurant, AI, ACL, AGNews Continual Learning Sequence (test); Restaurant MF1: 53.14.

- [End-task Classification on Continual DAP Domain Suite Restaurant, ACL, AI, Phone, Camera, PubMed](<https://www.sota2.com/research/sota/end-task-classification-on-continual-dap-domain-suite-restaurant-acl-ai-phone-camera-pubmed.md>): End-task Classification; Continual DAP Domain Suite Restaurant, ACL, AI, Phone, Camera, PubMed; Restaurant Macro F1: 0.7999.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2108.05036>)

[PDF](<https://arxiv.org/pdf/2108.05036.pdf>)