# Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

Source: [Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision](<https://www.sota2.com/research/paper/69afa5662d603e6de3e43875>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-28T05:00:13.546Z

Authors: Shreyas Gopal, Donghang Wu, Ashutosh Anshul, Yeo Yue Heng, Yizhou Peng, Haoyang Li, Hexin Liu, Eng Siong Chng

Published: 2026-03-07T04:09:47.000Z

## Abstract

Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Open-ended Instruction Following on Audio-OpenHermes](<https://www.sota2.com/research/sota/open-ended-instruction-following-on-audio-openhermes.md>): Open-ended instruction following; Audio-OpenHermes; EN Score: 4.42.

- [Close-ended Spoken Question Answering on Audio-MLQA](<https://www.sota2.com/research/sota/close-ended-spoken-question-answering-on-audio-mlqa.md>): Close-ended Spoken Question Answering; Audio-MLQA; Score (EN): 4.16.

- [Open-ended Instruction Following on AlpacaEval Audio](<https://www.sota2.com/research/sota/open-ended-instruction-following-on-alpacaeval-audio.md>): Open-ended instruction following; AlpacaEval Audio; EN Score: 4.58.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.07025>)

[PDF](<https://arxiv.org/pdf/2603.07025.pdf>)