# Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition

Source: [Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition](<https://www.sota2.com/research/paper/69aa5f702d603e6de3e4180d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-07T05:00:04.447Z

Authors: Mengze Hong, Yi Gu, Di Jiang, Hanlin Gu, Chen Jason Zhang, Lu Wang, Zhiyang Su

Published: 2026-03-05T08:42:17.000Z

## Abstract

Training automatic speech recognition (ASR) models increasingly relies on decentralized federated learning to ensure data privacy and accessibility, producing multiple local models that require effective merging. In hybrid ASR systems, while acoustic models can be merged using established methods, the language model (LM) for rescoring the N-best speech recognition list faces challenges due to the heterogeneity of non-neural n-gram models and neural network models. This paper proposes a heterogeneous LM optimization task and introduces a match-and-merge paradigm with two algorithms: the Genetic Match-and-Merge Algorithm (GMMA), using genetic operations to evolve and pair LMs, and the Reinforced Match-and-Merge Algorithm (RMMA), leveraging reinforcement learning for efficient convergence. Experiments on seven OpenSLR datasets show RMMA achieves the lowest average Character Error Rate and better generalization than baselines, converging up to seven times faster than GMMA, highlighting the paradigm's potential for scalable, privacy-preserving ASR systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Automatic Speech Recognition on SLR93 (AISHELL-3) (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr93-aishell-3-test.md>): Automatic Speech Recognition; SLR93 (AISHELL-3) (test); CER: 4.55.

- [Automatic Speech Recognition on SLR62 aidatatang (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr62-aidatatang-test.md>): Automatic Speech Recognition; SLR62 aidatatang (test); CER: 4.14.

- [Automatic Speech Recognition on SLR18 (THCHS-30) (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr18-thchs-30-test.md>): Automatic Speech Recognition; SLR18 (THCHS-30) (test); CER: 13.68.

- [Automatic Speech Recognition on SLR68 Read Speech (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr68-read-speech-test.md>): Automatic Speech Recognition; SLR68 Read Speech (test); CER: 8.77.

- [Automatic Speech Recognition on SLR33 Aishell (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr33-aishell-test.md>): Automatic Speech Recognition; SLR33 Aishell (test); CER: 4.63.

- [Automatic Speech Recognition on SLR47 Primewords (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr47-primewords-test.md>): Automatic Speech Recognition; SLR47 Primewords (test); CER: 17.57.

- [Automatic Speech Recognition on SLR38 Free ST Mandarin (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-slr38-free-st-mandarin-test.md>): Automatic Speech Recognition; SLR38 Free ST Mandarin (test); CER: 9.26.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.04945>)

[PDF](<https://arxiv.org/pdf/2603.04945.pdf>)
