# Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Source: [Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS](<https://www.sota2.com/research/paper/69cf49980636e5bf4afcda63>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-04T05:00:04.515Z

Authors: Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov, Nikita Vasiliev, Mikhail Gorodnichev, Grach Mkrtchian

Published: 2026-03-31T21:47:52.000Z

## Abstract

We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-to-Speech on Balalaika corpus (test)](<https://www.sota2.com/research/sota/text-to-speech-on-balalaika-corpus-test.md>): Text-to-Speech; Balalaika corpus (test); MOS (Mean Opinion Score): 2.521.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.01247>)

[PDF](<https://arxiv.org/pdf/2604.01247.pdf>)
