Video-to-Audio Generation on VGGSound original (test)
62.37Inception ScoreDIFF-FOLEY
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| DIFF-FOLEYVisual Feature=CAVP, FPS=4, Guidance=Double Guidance, Inference Time=0.38s2023.06 | 62.37 | 9.87 | 6.43 | 94.05 | — | — | — | |
| DIFF-FOLEYVisual Feature=CAVP, FPS=4, Guidance=Classifier-Free Guidance (CFG), Inference Time=0.38s2023.06 | 53.34 | 11.22 | 6.36 | 92.67 | — | — | — | |
| Im2WavVisual Feature=CLIP, FPS=30, Guidance=Classifier-Free Guidance (CFG), Inference Time=6.41s2023.06 | 39.3 | 11.44 | 5.2 | 67.4 | — | — | — | |
| SpecVQGANVisual Feature=ResNet50, FPS=21.5, Guidance=None, Inference Time=5.47s2023.06 | 30.8 | 9.7 | 7.03 | 49.19 | — | — | — | |
| SpecVQGANVisual Feature=RGB + Flow, FPS=21.5, Guidance=None, Inference Time=5.47s2023.06 | 30.01 | 8.93 | 6.93 | 52.94 | — | — | — | |
| Diffusion Latent Alignervariant=full, denoising_steps=302024.02 | 5.831 | 32.92 | 2.619 | — | — | — | 7.316 | |
| Diffusion Latent Alignervariant=vanilla, denoising_steps=302024.02 | 5.625 | 40.457 | 3.203 | — | — | — | 6.85 | |
| SpecVQGANbackbone=ResNet-50, training_dataset=VGGSound2024.02 | 5.108 | 37.269 | 3.29 | — | — | — | 7.736 | |
| CLIPSonic-IQ2023.08 | — | — | — | — | 2.533 | 2.14 | — | |
| Im2Wav2023.08 | — | — | — | — | 1.838 | 2.415 | — | |
| Reference2023.08 | — | — | — | — | 3.58 | 4.178 | — | |
| V2A-MapperCLIP backbone=ViT-B/32, Audio generator=AudioLDM2023.08 | — | — | — | — | 2.845 | 2.808 | — |