# Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling Source: [Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling]() Collection index: [llms.txt]() Data updated: 2026-04-03T05:00:04.568Z Authors: Kazuki Yano, Jun Suzuki, Shinji Watanabe Published: 2026-04-01T05:16:06.000Z ## Abstract Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Zero-shot Text Evaluation on ARC-e, ARC-c, BoolQ, HSwag, OBQA, PIQA, WinoGrande, MMLU](): Zero-shot Text Evaluation; Text Benchmarks ARC-e, ARC-c, BoolQ, HS, OBQA, PIQA, WG, MMLU; Average Accuracy: 66.4. - [Automatic Speech Recognition on LibriSpeech clean (test)](): Automatic Speech Recognition; LibriSpeech clean (test); WER: 2.4. ## Original sources and implementation [Original paper]() [PDF]()