# Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition

Source: [Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition](<https://www.sota2.com/research/paper/68fa738a1e209131cea5d173>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T06:20:43.969Z

Authors: Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, Kei Sawada

Published: 2023-12-06T18:34:42.000Z

## Abstract

Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Automatic Speech Recognition on ReazonSpeech (test)](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-reazonspeech-test.md>): ASR; ReazonSpeech (test); CER: 8.4.

- [ASR on CV 8.0 (test)](<https://www.sota2.com/research/sota/asr-on-cv-8-0-test.md>): ASR; CV 8.0 (test); CER: 9.1.

- [Automatic Speech Recognition on JSUT basic5000](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-jsut-basic5000.md>): ASR; JSUT basic5000; CER: 8.6.

- [Automatic Speech Recognition on CSJ](<https://www.sota2.com/research/sota/automatic-speech-recognition-on-csj.md>): ASR; CSJ; Eval1 Error Rate: 0.309.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.03668>)

[PDF](<https://arxiv.org/pdf/2312.03668.pdf>)

[Implementation](<https://huggingface.co/rinna/nue-asr>)
