# Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models

Source: [Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models](<https://www.sota2.com/research/paper/68fa7c1f1e209131cea8c83d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:57:13.861Z

Authors: Tsung-En Lin, Kuan-Yi Lee, Hung-Yi Lee

Published: 2025-10-14T08:52:18.000Z

## Abstract

Large Audio-Language Models and Multi-Modal Large Language Models have demonstrated strong capabilities in tasks such as Audio Question Answering (AQA), Audio Captioning, and Automatic Speech Recognition (ASR). However, there is growing evidence that these models can hallucinate about the content of the audio. To address this issue, we probe the models' internal states and propose Adaptive Vector Steering (AVS), a method that better grounds generation in audio content. We also identify a strong correlation between output correctness and internal representations. Experiments show consistent performance gains across two models and two benchmarks. On the Audio Hallucination QA dataset, our method boosts the F1-score of Gemma from 0.550 to 0.619 and Qwen from 0.626 to 0.632. Furthermore, our method increases the accuracy of Qwen on MMAU from 0.548 to 0.592, marking an 8% relative increase. To the best of our knowledge, this is the first work to apply vector steering to mitigate hallucination in audio.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.12851>)

[PDF](<https://arxiv.org/pdf/2510.12851.pdf>)
