# MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free

Source: [MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free](<https://www.sota2.com/research/paper/695de86953fdd5c361743aae>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-10T05:00:05.200Z

Authors: Yishu Lei, Shuwei He, Jing Hu, Dan Zhang, Xianlong Luo, Danxiang Zhu, Shikun Feng, Rui Liu, Jingzhou He, Yu Sun, Hua Wu, Haifeng Wang

Published: 2026-01-06T12:24:38.000Z

## Abstract

Extending the input modality of Large Language Models~(LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically \\textit{heterogeneous}, entangling attributes such as speech, music, and environmental context. Existing research is limited to a dense, parameter-shared adapter to model these diverse patterns, which induces \\textit{gradient conflict} during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the \\textit{\\textbf{MoE-Adapter}}, a sparse Mixture-of-Experts~(MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts capturing complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. Furthermore, we will release the related code and models to facilitate future research.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Automatic Speech Recognition on OGI-S (Age 11-15)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-ogi-s-age-11-15.md>): Automatic Speech Recognition; OGI-S (Age 11-15); WER: 9.26.

- [Automatic Speech Recognition on OGI-S (Age 4-7)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-ogi-s-age-4-7.md>): Automatic Speech Recognition; OGI-S (Age 4-7); WER: 19.55.

- [Automatic Speech Recognition on OGI-S Age 8-10](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-ogi-s-age-8-10.md>): Automatic Speech Recognition; OGI-S Age 8-10; WER: 10.81.

- [Automatic Speech Recognition on OGI-S (Avg.)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-ogi-s-avg.md>): Automatic Speech Recognition; OGI-S (Avg.); WER: 12.08.

- [Automatic Speech Recognition on MyST (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-myst-test.md>): Automatic Speech Recognition; MyST (test); WER: 8.63.

- [Automatic Speech Recognition on LibriSpeech clean (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-librispeech-clean-test.md>): Automatic Speech Recognition; LibriSpeech clean (test); WER: 1.96.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.02967>)

[PDF](<https://arxiv.org/pdf/2601.02967.pdf>)
