Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition
About
Serialized output training (SOT) attracts increasing attention due to its convenience and flexibility for multi-speaker automatic speech recognition (ASR). However, it is not easy to train with attention loss only. In this paper, we propose the overlapped encoding separation (EncSep) to fully utilize the benefits of the connectionist temporal classification (CTC) and attention hybrid loss. This additional separator is inserted after the encoder to extract the multi-speaker information with CTC losses. Furthermore, we propose the serialized speech information guidance SOT (GEncSep) to further utilize the separated encodings. The separated streams are concatenated to provide single-speaker information to guide attention during decoding. The experimental results on LibriMix show that the single-speaker encoding can be separated from the overlapped encoding. The CTC loss helps to improve the encoder representation under complex scenarios. GEncSep further improved performance.
Key insights
How does this paper improve multi-speaker ASR training?
SOT-based ASR struggles to use CTC loss because serialized labels are hard to align with overlapped speech. The paper adds an encoder separator (EncSep) for CTC training and concatenates separated streams to guide decoding (GEncSep).
What distinguishes EncSep and GEncSep from prior SOT methods?
Unlike SOT baselines using only attention loss, EncSep adds a separator for CTC without extra decoding cost. On noisy Libri2Mix/Libri3Mix eval sets, GEncSep achieved over 12% and 9% relative WER improvement versus SOT.
What practical use do these findings have?
The methods improve multi-speaker ASR in noisy and three-speaker conditions, evaluated on Libri2Mix and Libri3Mix benchmarks. No production deployment is documented; results are benchmark prototypes using Conformer and WavLM features.
What limitations and future work do the authors note?
The proposed end-to-end method still trails pipeline systems with separation front-ends (e.g., TSE-Whisper) on noisy two-speaker conditions. Future work involves fusing overlapped and separated embeddings to use different information sources.