# An Exploration of Mamba for Speech Self-Supervised Models

Source: [An Exploration of Mamba for Speech Self-Supervised Models](<https://www.sota2.com/research/paper/68fa79681e209131cea7d72e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-22T05:23:53.566Z

Authors: Tzu-Quan Lin, Heng-Cheng Kuo, Tzu-Chieh Wei, Hsi-Chun Cheng, Chun Wei Chen, Hsien-Fu Hsiao, Yu Tsao, Hung-yi Lee

Published: 2025-06-14T19:00:44.000Z

## Abstract

While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised learning (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction. The codebase is available at https://github.com/hckuo145/Mamba-based-HuBERT.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speech Probing on SUPERB](<https://www.sota2.com/research/sota/speech-probing-on-superb.md>): Speech Probing; SUPERB; PR Error Rate: 10.65.

- [Universal Speech Representation Evaluation on SUPERB Benchmark](<https://www.sota2.com/research/sota/universal-speech-representation-evaluation-on-superb-benchmark.md>): Universal Speech Representation Evaluation; SUPERB Benchmark; Overall Score: 823.15.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.12606>)

[PDF](<https://arxiv.org/pdf/2506.12606.pdf>)