# Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models

Source: [Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models](<https://www.sota2.com/research/paper/68fa78c11e209131cea7a528>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:16.911Z

Authors: Hao Yang, Lizhen Qu, Ehsan Shareghi, Gholamreza Haffari

Published: 2025-05-26T08:25:25.000Z

## Abstract

Large Audio Language Models (LALMs) have extended the capabilities of Large Language Models (LLMs) by enabling audio-based human interactions. However, recent research has revealed that LALMs remain vulnerable to harmful queries due to insufficient safety-alignment. Despite advances in defence measures for text and vision LLMs, effective safety-alignment strategies and audio-safety dataset specifically targeting LALMs are notably absent. Meanwhile defence measures based on Supervised Fine-tuning (SFT) struggle to address safety improvement while avoiding over-rejection issues, significantly compromising helpfulness. In this work, we propose an unsupervised safety-fine-tuning strategy as remedy that reshapes model's representation space to enhance existing LALMs safety-alignment while balancing the risk of over-rejection. Our experiments, conducted across three generations of Qwen LALMs, demonstrate that our approach significantly improves LALMs safety under three modality input conditions (audio-text, text-only, and audio-only) while increasing over-rejection rate by only 0.88% on average. Warning: this paper contains harmful examples.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Helpfulness Assessment on AdvBench-audio Safe](<https://www.sota2.com/research/sota/helpfulness-assessment-on-advbench-audio-safe.md>): Helpfulness Assessment; AdvBench-audio Safe; BRR: 86.44.

- [Harmfulness Assessment on AJailBench](<https://www.sota2.com/research/sota/harmfulness-assessment-on-ajailbench.md>): Harmfulness Assessment; AJailBench; ASR: 54.

- [Harmfulness Evaluation on Figstep-audio Harmful](<https://www.sota2.com/research/sota/harmfulness-evaluation-on-figstep-audio-harmful.md>): Harmfulness Evaluation; Figstep-audio Harmful; ASR: 52.4.

- [Harmfulness Evaluation on SORRY-Bench audio](<https://www.sota2.com/research/sota/harmfulness-evaluation-on-sorry-bench-audio.md>): Harmfulness Evaluation; SORRY-Bench audio; ASR Accuracy: 38.41.

- [Harmfulness Evaluation on AdvBench-audio Harmful](<https://www.sota2.com/research/sota/harmfulness-evaluation-on-advbench-audio-harmful.md>): Harmfulness Evaluation; AdvBench-audio Harmful; ASR Score: 3.27.

- [Helpfulness Evaluation on Figstep-audio Harmful-Safe](<https://www.sota2.com/research/sota/helpfulness-evaluation-on-figstep-audio-harmful-safe.md>): Helpfulness evaluation; Figstep-audio Harmful-Safe; BRR: 70.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.19670>)

[PDF](<https://arxiv.org/pdf/2505.19670.pdf>)