Spoken Dialogue on MultiDialog 1.0 (test)
930.401PPLSpeechGPT
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SpeechGPTInput Modality=A, Output Modality=A2024.06 | 930.401 | 20.536 | 0.064 | 0.054 | 0.743 | 0.876 | |
| GT AV Speech Token2024.06 | 1,054.643 | 76.326 | 0.565 | 0.474 | 0.947 | 0.996 | |
| Audio-Visual Spoken Dialogue System (+ AVSR/TTS Pretraining)Input Modality=AV, Output Modality=AV, Pretraining=AVSR/TTS2024.06 | 1,068.904 | 22.09 | 0.062 | 0.066 | 0.943 | 0.965 | |
| d-GSLMInput Modality=A, Output Modality=A2024.06 | 1,085.265 | 8.197 | 0.065 | 0.064 | 0.883 | 0.876 | |
| AVSR + LM + TTS + TFGInput Modality=AV, Output Modality=AV2024.06 | 1,157.586 | 47.287 | 0.075 | 0.1 | 0.959 | 0.977 | |
| Audio-Visual Spoken Dialogue System (+ LLM initialized)Input Modality=AV, Output Modality=AV, Initialization=LLM pretrained2024.06 | 1,237.757 | 17.098 | 0.059 | 0.058 | 0.936 | 0.963 | |
| Audio-Visual Spoken Dialogue System (+ Mixed Text-AV Speech Pretraining)Input Modality=AV, Output Modality=AV, Pretraining=Mixed Text-AV2024.06 | 1,248.001 | 24.094 | 0.063 | 0.065 | 0.945 | 0.957 | |
| Audio-Visual Spoken Dialogue System (Scratch)Input Modality=AV, Output Modality=AV, Initialization=Scratch2024.06 | 1,898.864 | 13.305 | 0.058 | 0.064 | 0.945 | 0.955 |