Audio Captioning on Clotho
50.9CIDErChen et al.
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Chen et al.2024.09 | 50.9 | 19.4 | 14.5 | 32.7 | 54.1 | — | — | — | — | — | — | — | — | — | — | |
| Feng et al.2024.09 | 49.5 | 19.2 | 14.1 | 31.8 | 52.5 | — | — | — | — | — | — | — | — | — | — | |
| Liu et al.2024.09 | 49.3 | 19.5 | 14.5 | 31.9 | 53.3 | — | — | — | — | — | — | — | — | — | — | |
| CNN14-BART2025.03 | 48.8 | 18.5 | 13.3 | 31 | — | — | — | — | — | — | — | — | — | — | — | |
| CNN14-BART2025.03 | 48.8 | 18.5 | 13.3 | 31 | — | — | — | — | — | — | — | — | — | — | — | |
| Wavcaps0-shot=false2024.06 | 48.8 | — | — | 31 | — | — | — | — | — | — | — | — | — | — | — | |
| EnCLAP++2024.09 | 48 | 19.9 | 14.8 | 31.4 | 54.4 | — | — | — | — | — | — | — | — | — | — | |
| EnCLAP-largetraining_setup=Trained on AudioCaps and Clotho2024.01 | 46.4 | 18.6 | 13.3 | 29.9 | — | — | — | — | — | — | — | — | — | — | — | |
| EnCLAP-basetraining_setup=Trained on AudioCaps and Clotho2024.01 | 46.3 | 18.4 | 12.8 | 29.5 | — | — | — | — | — | — | — | — | — | — | — | |
| HTSAT-BART2025.03 | 46.2 | 18.4 | 13.3 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | |
| HTSAT-BART2025.03 | 46.2 | 18.4 | 13.3 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | |
| EnCLAP-basetraining_setup=Trained on Clotho only2024.01 | 46.1 | 18 | 12.8 | 29.4 | — | — | — | — | — | — | — | — | — | — | — | |
| DCASE 2024 Baseline2024.09 | 44.2 | 18.6 | 13.5 | 28.8 | 51 | — | — | — | — | — | — | — | — | — | — | |
| Qwen-Audiozero-shot=true2023.11 | 44.1 | — | 13.6 | 28.8 | — | — | — | — | — | — | — | — | — | — | — | |
| GraphACtraining_setup=Trained on AudioCaps and Clotho2024.01 | 43.7 | 17.5 | 12.6 | 28.1 | — | — | — | — | — | — | — | — | — | — | — | |
| FeatureCut0-shot=false2024.06 | 43.6 | — | — | 27.9 | — | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP stage2Audio encoder AudioSet fine-tuning=false2025.03 | 43.5 | 17.8 | 12.4 | 28 | 49.9 | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP2025 stage2Audio encoder AudioSet fine-tuning=false2025.03 | 43.5 | 17.8 | 12.4 | 28 | 49.9 | — | — | — | — | — | — | — | — | — | — | |
| P-LocalAFTtraining_setup=Trained on AudioCaps and Clotho2024.01 | 43.4 | 17.7 | 12.2 | 27.8 | — | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP stage2.1Audio encoder AudioSet fine-tuning=false2025.03 | 43.2 | 17.8 | 12.5 | 27.8 | 49.7 | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP2025 stage2.1Audio encoder AudioSet fine-tuning=false2025.03 | 43.2 | 17.8 | 12.5 | 27.8 | 49.7 | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP2025Audio encoder AudioSet fine-tuning=true2025.03 | 43.1 | 17.7 | 12.5 | 27.8 | 49.6 | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP2025Audio encoder AudioSet fine-tuning=true2025.03 | 43.1 | 17.7 | 12.5 | 27.8 | 49.6 | — | — | — | — | — | — | — | — | — | — | |
| EnCLAP-largetraining_setup=Trained on Clotho only2024.01 | 42.6 | 18.2 | 12.9 | 27.8 | — | — | — | — | — | — | — | — | — | — | — | |
| CLAP2023Audio encoder AudioSet fine-tuning=true2025.03 | 42.5 | 17.5 | 12.3 | 27.4 | 49.2 | — | — | — | — | — | — | — | — | — | — | |
| CLAP2023Audio encoder AudioSet fine-tuning=false2025.03 | 42.5 | 17.5 | 12.3 | 27.4 | 49.2 | — | — | — | — | — | — | — | — | — | — | |
| MAACtraining_setup=Trained on Clotho only2024.01 | 41.9 | 17.4 | 11.9 | 26.9 | — | — | — | — | — | — | — | — | — | — | — | |
| PengiTask-specific fine-tuning=true2023.11 | 41.6 | — | 12.6 | 27.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Pengitraining_setup=Trained on AudioCaps and Clotho2024.01 | 41.6 | 17.2 | 12.6 | 27.1 | — | — | — | — | — | — | — | — | — | — | — | |
| LAION-CLAPAudio encoder AudioSet fine-tuning=true2025.03 | 41.4 | 17.4 | 12.1 | 26.8 | 48.6 | — | — | — | — | — | — | — | — | — | — | |
| LAION-CLAPAudio encoder AudioSet fine-tuning=true2025.03 | 41.4 | 17.4 | 12.1 | 26.8 | 48.6 | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP stage1Audio encoder AudioSet fine-tuning=false2025.03 | 41.2 | 17.4 | 12.2 | 26.7 | 48.9 | — | — | — | — | — | — | — | — | — | — | |
| M2D-CLAP2025 stage1Audio encoder AudioSet fine-tuning=false2025.03 | 41.2 | 17.4 | 12.2 | 26.7 | 48.9 | — | — | — | — | — | — | — | — | — | — | |
| Kim et al.2024.09 | 40.9 | 18.9 | 13.5 | 27.2 | 52.6 | — | — | — | — | — | — | — | — | — | — | |
| LHDFFtraining_setup=Trained on Clotho only2024.01 | 40.8 | 17.5 | 12.2 | 26.5 | — | — | — | — | — | — | — | — | — | — | — | |
| CLIP-AACtraining_setup=Trained on Clotho only2024.01 | 40.7 | 17.1 | 11.9 | 26.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Prefix Tuning for Automated Audio CaptioningTraining set=Clotho2023.03 | 39.2 | 17 | 11.8 | 25.5 | — | 56 | 37.6 | 25.3 | 16 | 37.8 | — | — | — | — | — | |
| Individuals2026.05 | 38.63 | — | 11.98 | 25.29 | — | — | — | — | — | — | — | — | — | — | — | |
| DiDi-Merging2026.05 | 38.56 | — | 11.98 | 25.25 | — | — | — | — | — | — | — | — | — | — | — | |
| FREE-Merging2026.05 | 38.53 | — | 11.95 | 25.22 | — | — | — | — | — | — | — | — | — | — | — | |
| MMER2026.05 | 38.49 | — | 11.93 | 25.18 | — | — | — | — | — | — | — | — | — | — | — | |
| Koh et al.Training set=Clotho2023.03 | 38 | 16.5 | 11.1 | 24.6 | — | 55.1 | 36.9 | 25.2 | 16.8 | 37.3 | — | — | — | — | — | |
| Xu et al.Training set=Clotho2023.03 | 37.7 | 16.9 | 11.5 | 24.6 | — | 55.6 | 36.3 | 24.2 | 15.9 | 36.8 | — | — | — | — | — | |
| NaiveMC2026.05 | 37.56 | — | 11.61 | 24.61 | — | — | — | — | — | — | — | — | — | — | — | |
| Chen et al.Training set=Clotho2023.03 | 34.6 | 16 | 10.8 | 22.7 | — | 53.4 | 34.3 | 23 | 15.1 | 35.6 | — | — | — | — | — | |
| Jung et al.2024.09 | 34.4 | 17.2 | 14 | 24.2 | 55.4 | — | — | — | — | — | — | — | — | — | — | |
| Cacophony2025.03 | 34.2 | 15.3 | 10.6 | 22.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Cacophony2025.03 | 34.2 | 15.3 | 10.6 | 22.4 | — | — | — | — | — | — | — | — | — | — | — | |
| MiCo-Chat-7B0-shot=true2024.06 | 33.3 | — | — | 21.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Mei et al.Training set=Clotho2023.03 | 32 | 15.8 | 10.5 | 21.3 | — | 52.7 | 32.7 | 21.1 | 13.1 | 35.6 | — | — | — | — | — | |
| Prefix Tuning for Automated Audio CaptioningTraining set=Clotho & AudioCaps2023.03 | 31.9 | 15.9 | 11.1 | 21.5 | — | 53.9 | 34.6 | 22.7 | 14.2 | 36.6 | — | — | — | — | — | |
| Chen et al.Training set=Clotho & AudioCaps2023.03 | 31.4 | 15.3 | 10.2 | 20.8 | — | 51.6 | 32.5 | 21.5 | 14.1 | 35 | — | — | — | — | — | |
| Mei et al.Training set=Clotho & AudioCaps2023.03 | 31.3 | 15.7 | 10.5 | 20.9 | — | 51.6 | 31.8 | 20.4 | 12.7 | 35.1 | — | — | — | — | — | |
| OneLLM-7B0-shot=true2024.06 | 29.1 | — | — | 19.5 | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineKeep Ratio=0.752026.05 | 28.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineKeep Ratio=0.52026.05 | 28.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineKeep Ratio=0.252026.05 | 28.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gontier et al.Training set=Clotho2023.03 | 27.8 | 14.8 | 9.2 | 18.5 | — | 50.6 | 31.8 | 21 | 13.4 | 33.8 | — | — | — | — | — | |
| ChatBridge-13B0-shot=true2024.06 | 26.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gontier et al.Training set=Clotho & AudioCaps2023.03 | 25.1 | 13.6 | 8.3 | 16.7 | — | 46.1 | 28.2 | 18.2 | 11.7 | 31.8 | — | — | — | — | — | |
| Prefix Tuning for Automated Audio CaptioningTraining set=AudioCaps2023.03 | 19.2 | 11.2 | 7.4 | 13.3 | — | 34.2 | 19.5 | 11.5 | 6.5 | 27.6 | — | — | — | — | — | |
| LTBM (local)Keep Ratio=0.752026.05 | 18.38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Global MergeKeep Ratio=0.752026.05 | 18.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RNN-GRU-EncDec + VGGish + Word2VecInput Features=VGGish, Word Embeddings=Word2Vec2020.06 | 18 | 19 | — | — | — | 51 | 28 | 22 | 12 | 40 | — | — | — | — | — | |
| Segmentwise Top-KKeep Ratio=0.752026.05 | 16.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Global Top-KKeep Ratio=0.752026.05 | 15.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RNN-GRU-EncDec + VGGishInput Features=VGGish2020.06 | 15 | 18 | — | — | — | 45 | 24 | 18 | 9 | 38 | — | — | — | — | — | |
| Mei et al.Training set=AudioCaps2023.03 | 11.7 | 9.6 | 5 | 8.4 | — | 29.4 | 14.6 | 8 | 4.3 | 23.9 | — | — | — | — | — | |
| Global MergeKeep Ratio=0.52026.05 | 11.38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gontier et al.Training set=AudioCaps2023.03 | 11.2 | 9.8 | 4.6 | 7.9 | — | 30.9 | 14.6 | 7.1 | 3.4 | 23.3 | — | — | — | — | — | |
| RNN-GRU-EncDec + Log Mel EnergyInput Features=Log Mel Energy2020.06 | 11 | 17 | — | — | — | 45 | 21 | 16 | 8 | 34 | — | — | — | — | — | |
| LTBM (local)Keep Ratio=0.52026.05 | 10.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Chen et al.Training set=AudioCaps2023.03 | 10.9 | 8.6 | 4.2 | 7.6 | — | 22.6 | 11.4 | 6.5 | 3.9 | 22.8 | — | — | — | — | — | |
| Clotho2020.06 | 10 | 9 | — | — | — | 42 | 14 | 6 | 2 | 27 | — | — | — | — | — | |
| UniAvgKeep Ratio=0.52026.05 | 9.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Segmentwise Top-KKeep Ratio=0.52026.05 | 9.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Global Top-KKeep Ratio=0.52026.05 | 9.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LTBM (local)Keep Ratio=0.252026.05 | 6.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Global MergeKeep Ratio=0.252026.05 | 6.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Global Top-KKeep Ratio=0.252026.05 | 6.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Segmentwise Top-KKeep Ratio=0.252026.05 | 5.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniAvgKeep Ratio=0.252026.05 | 4.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLAP2023 (Ref)2025.03 | — | — | — | 27.1 | — | — | — | — | — | — | — | — | — | — | — | |
| CLAP2023 (Reference)2025.03 | — | — | — | 27.1 | — | — | — | — | — | — | — | — | — | — | — | |
| GAMA-7Br=8, Conn.=QFormer2025.09 | — | — | — | — | — | — | — | — | — | — | 13.5 | — | — | — | — | |
| LoRA-MCLepsilon=0.0005, decoding=BS, beam=52025.07 | — | 24.4 | 19.6 | 43.6 | — | 68.7 | — | — | 15.6 | 47.6 | — | 58.1 | 48.8 | 59.9 | 70 | |
| LoRA-MCLepsilon=0.05, decoding=BS, beam=52025.07 | — | 24.9 | 19.6 | 43.4 | — | 68.9 | — | — | 16.2 | 48 | — | 58.1 | 47.8 | 59.9 | 69.7 | |
| LoRA-MCLmode=annealed, decoding=BS, beam=52025.07 | — | 24.6 | 19.9 | 44.3 | — | 68.9 | — | — | 16.8 | 47.7 | — | 58.4 | 47.8 | 60 | 71.1 | |
| LoRA-MCLmode=annealed, decoding=Nucleus (Top-p) sampling2025.07 | — | 21.2 | 17.1 | 32.5 | — | — | — | — | 7.4 | 40.3 | — | 82 | 9.8 | 58.9 | 50.7 | |
| LoRA-MCL (annealed)Decoding=Top-k sampling2025.07 | — | 21.2 | 17.5 | 32.7 | — | — | — | — | 7.4 | 40.3 | — | 81.9 | 9.7 | 59.7 | 50.7 | |
| LoRA-MCL (annealed)Decoding=Nucleus (Top-p) sampling2025.07 | — | 21.2 | 17.1 | 32.5 | — | — | — | — | 7.4 | 40.3 | — | 82 | 9.8 | 58.9 | 50.7 | |
| LoRA-MCL (annealed)Decoding=Typical p sampling2025.07 | — | 21.1 | 17.2 | 32.7 | — | — | — | — | 7.2 | 40.2 | — | 81.9 | 10.3 | 59 | 50.9 | |
| LoRA-MLErank=8, decoding=BS, beam=52025.07 | — | 22.8 | 17.4 | 39.4 | — | 65.6 | — | — | 13.7 | 44.5 | — | 36.5 | 82.2 | 57.5 | 62.6 | |
| LoRA-MLErank=40, decoding=BS, beam=52025.07 | — | 22.6 | 17.2 | 37.6 | — | 64.3 | — | — | 11.5 | 43.5 | — | 36.7 | 81.8 | 57 | 59.5 | |
| LoRA-MLErank=8, decoding=DBS (λ = 1.0), beam=52025.07 | — | 24.2 | 19.4 | 42.5 | — | 68.5 | — | — | 14.8 | 47 | — | 62.5 | 41.7 | 60.2 | 68.1 | |
| LoRA-MLErank=8, decoding=Top-k sampling2025.07 | — | 21.5 | 17.6 | 32.3 | — | — | — | — | 6.6 | 40.2 | — | 81.3 | 10.9 | 59.3 | 49.7 | |
| LoRA-MLE (r = 8)Decoding=Top-k sampling2025.07 | — | 21.5 | 17.6 | 32.3 | — | — | — | — | 6.6 | 40.2 | — | 81.3 | 10.9 | 59.3 | 49.7 | |
| LoRA-MLE (r = 8)Decoding=Nucleus (Top-p) sampling2025.07 | — | 21.2 | 17.3 | 33 | — | — | — | — | 7.3 | 40.3 | — | 81.2 | 11.2 | 58.6 | 51.6 | |
| LoRA-MLE (r = 8)Decoding=Typical p sampling2025.07 | — | 21.2 | 17.3 | 33.1 | — | — | — | — | 7.3 | 40.3 | — | 81.2 | 11.1 | 58.6 | 51.6 | |
| LoRA-MoEdecoding=BS, beam=252025.07 | — | 23.9 | 18.4 | 41.1 | — | 65.7 | — | — | 14.4 | 45.7 | — | 41 | 75.5 | 57.9 | 65.3 |