Turn-taking and backchannel prediction on MM-F2F
82.3AccuracyMulti-Modal Framework (Ours)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Multi-Modal Framework (Ours)Modalities=Text + Audio + Video (T+A+V)2025.05 | 82.3 | 80.6 | 81.1 | 0.906 | |
| MM-F2FModal=T+A+V2026.03 | 82.3 | 80.6 | 81.1 | 90.6 | |
| RESPONDModal=T2026.03 | 75.6 | 77 | 77.3 | 69.7 | |
| MM-F2FModal=T2026.03 | 75.1 | 74.7 | 76.7 | 70.7 | |
| Wang et al.Modalities=Text + Audio (T+A)2025.05 | 73.7 | 74.2 | 73.9 | 0.68 | |
| Wang et al.’sModal=T+A2026.03 | 73.7 | 74.2 | 73.9 | 68 | |
| Kurata et al.Modalities=Text + Audio + Video (T+A+V)2025.05 | 72 | 72.9 | 72.8 | 0.667 | |
| Kurata et al.’sModal=T+A+V2026.03 | 72 | 72.9 | 72.8 | 66.7 | |
| TurnGPTModalities=Text (T)2025.05 | 64.5 | 74.5 | 42 | — | |
| TurnGPTModal=T2026.03 | 64.5 | 74.5 | 42 | — |