# Targeted Speaker Poisoning Framework in Zero-Shot Text-to-Speech

Source: [Targeted Speaker Poisoning Framework in Zero-Shot Text-to-Speech](<https://www.sota2.com/research/paper/69afa5a92d603e6de3e43d5b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:00:05.013Z

Authors: Thanapat Trachu, Thanathai Lertpetchpun, Sai Praneeth Karimireddy, Shrikanth Narayanan

Published: 2026-03-08T09:29:55.000Z

## Abstract

Zero-shot Text-to-Speech (TTS) voice cloning poses severe privacy risks, demanding the removal of specific speaker identities from trained TTS models. Conventional machine unlearning is insufficient in this context, as zero-shot TTS can dynamically reconstruct voices from just reference prompts. We formalize this task as Speech Generation Speaker Poisoning (SGSP), in which we modify trained models to prevent the generation of specific identities while preserving utility for other speakers. We evaluate inference-time filtering and parameter-modification baselines across 1, 15, and 100 forgotten speakers. Performance is assessed through the trade-off between utility (WER) and privacy, quantified using AUC and Forget Speaker Similarity (FSSIM). We achieve strong privacy for up to 15 speakers but reveal scalability limits at 100 speakers due to increased identity overlap. Our study thus introduces a novel problem and evaluation framework toward further advances in generative voice privacy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speaker Erasure on LibriTTS 1-speaker setting (retain test)](<https://www.sota2.com/research/sota/speaker-erasure-on-libritts-1-speaker-setting-retain-test.md>): Speaker Erasure; LibriTTS 1-speaker setting (retain test); WER: 2.69.

- [Speaker Erasure on LibriTTS 15 speakers forget set clean 1.0 (test)](<https://www.sota2.com/research/sota/speaker-erasure-on-libritts-15-speakers-forget-set-clean-1-0-test.md>): Speaker Erasure; LibriTTS 15 speakers forget set clean 1.0 (test); SSIM (R): 87.

- [Speaker Erasure on LibriTTS 100 speakers forget set 1.0 (test-clean)](<https://www.sota2.com/research/sota/speaker-erasure-on-libritts-100-speakers-forget-set-1-0-test-clean.md>): Speaker Erasure; LibriTTS 100 speakers forget set 1.0 (test-clean); SSIM (R): 83.

- [Speaker Erasure on LibriTTS 1-speaker setting (forget test)](<https://www.sota2.com/research/sota/speaker-erasure-on-libritts-1-speaker-setting-forget-test.md>): Speaker Erasure; LibriTTS 1-speaker setting (forget test); WER: 2.62.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.07551>)

[PDF](<https://arxiv.org/pdf/2603.07551.pdf>)