Audio Captioning on AudioCaps (test)
91.1CIDErLAMB
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LAMBPT Dataset=AC+CL+WC2026.01 | 91.1 | 19.7 | 55.4 | — | — | — | — | — | 27.1 | 67.7 | — | — | — | 55.3 | — | — | — | |
| LAVCap2025.01 | 84.9 | 18.5 | 51.7 | 72.3 | — | — | 29.7 | 51 | 26.2 | — | — | — | — | — | — | — | — | |
| MQ-CapPT Dataset=AC+CL+WC+AA2026.01 | 84.5 | 19.4 | 51.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SLAM-AACModel Category=Specialist, Parameters=7B, fine-tuned=true2024.12 | 84.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SLAM-AACPT Dataset=AC+CLP +WC +MA2026.01 | 84.1 | 19.4 | 51.8 | — | — | — | — | — | 26.8 | 66.8 | — | — | — | 51.5 | — | — | — | |
| AudioSetCaps2026.01 | 83.9 | 18.6 | — | — | — | — | — | — | 26.2 | — | — | — | — | — | — | — | — | |
| AutoCap (audio+text)Pre-training Dataset=AC+CL+WC2025.01 | 83.2 | 18.2 | 50.7 | 72.3 | — | — | 29.7 | 51.8 | 25.3 | — | — | — | — | — | — | — | — | |
| AutoCapInput Modality=audio+text, Pretraining Data=AC+CL+WC2024.06 | 83.2 | 18.2 | 50.7 | 72.3 | — | — | 29.7 | 51.8 | 25.3 | — | — | — | — | — | — | — | — | |
| LOAEPre-training Dataset=AC+CL+WC2025.01 | 81.6 | 19.3 | 50.5 | — | — | — | — | — | 26.7 | — | — | — | — | — | — | — | — | |
| LOAEPT Dataset=AC+CL+WC2026.01 | 81.6 | 19.3 | 50.5 | — | — | — | — | — | 26.7 | 66.4 | — | — | — | 50.4 | — | — | — | |
| CLAP-ARTPT Dataset=- / AC2026.01 | 80.7 | 18.8 | 49.8 | — | — | — | — | — | 25.6 | 65.5 | — | — | — | — | — | — | — | |
| CNext-transPre-training Dataset=AC+CL+MA+WC2025.01 | 80.6 | 18.4 | 49.5 | — | — | — | — | — | 25.2 | — | — | — | — | — | — | — | — | |
| CNext-transPretraining Data=AC+CL+MA+WC2024.06 | 80.6 | 18.4 | 49.5 | — | — | — | — | — | 25.2 | — | — | — | — | — | — | — | — | |
| AutoCapInput Modality=audio, Pretraining Data=AC+CL+WC2024.06 | 80.4 | 19 | 49.7 | 73.1 | — | — | 28.1 | 52 | 25.6 | — | — | — | — | — | — | — | — | |
| AutoCapPT Dataset=AC+CL+WC2026.01 | 80.4 | 19 | 49.7 | — | — | — | — | — | 25.6 | — | — | — | — | — | — | — | — | |
| EnCLAP-large2025.01 | 80.3 | 18.8 | 49.5 | — | — | — | — | — | 25.5 | — | — | — | — | — | — | — | — | |
| EnCLAP-largePT Dataset=AC+CL2026.01 | 80.3 | 18.8 | 49.5 | — | — | — | — | — | 25.5 | 65.5 | — | — | — | 49.9 | — | — | — | |
| ENCLAP-LargePretraining Data=None2024.06 | 80.2 | 18.8 | 49.5 | — | — | — | — | — | 25.5 | — | — | — | — | — | — | — | — | |
| AutoCap (audio+text)Pre-training Dataset=AC2025.01 | 80 | 18.8 | 49.4 | 72.1 | — | — | 28.6 | 51.5 | 25.6 | — | — | — | — | — | — | — | — | |
| AutoCapInput Modality=audio+text, Pretraining Data=AudioCaps (AC)2024.06 | 80 | 18.8 | 49.4 | 72.1 | — | — | 28.6 | 51.5 | 25.6 | — | — | — | — | — | — | — | — | |
| HTSAT-BARTSetting=Supervised2023.11 | 78.7 | 18.2 | 48.5 | — | — | — | 28.3 | 50.7 | 25 | — | — | — | — | — | — | — | — | |
| HTSAT-BARTPre-training Dataset=AC+CL+WC2025.01 | 78.7 | 18.2 | 48.5 | 70.7 | — | — | 28.3 | 50.7 | 25 | — | — | — | — | — | — | — | — | |
| HTSAT-BARTPretraining Data=AC+CL+WC2024.06 | 78.7 | 18.2 | 48.5 | 70.7 | — | — | 28.3 | 50.7 | 25 | — | — | — | — | — | — | — | — | |
| HTSAT-BARTPT Dataset=AC+CL+WC2026.01 | 78.7 | 18.2 | 48.5 | — | — | — | — | — | 25 | 64.2 | — | — | — | 48.3 | — | — | — | |
| AnyMALZeroshot=true, LLM Size=70B, Audio Encoder=CLAP2023.09 | 77.8 | 23 | 50.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoCapInput Modality=audio, Pretraining Data=AudioCaps (AC)2024.06 | 77.3 | 18.2 | 47.8 | 70 | — | — | 28 | 51.7 | 24.6 | — | — | — | — | — | — | — | — | |
| Multi-TTAPairMix=true, Multi-TTA=true2022.10 | 76.9 | 18.1 | 47.5 | 70 | 53.4 | 39.5 | 28.9 | 50.2 | 24.2 | — | — | — | — | — | — | — | — | |
| AL-MixGENPretraining Data=None2024.06 | 76.9 | 18.1 | 47.5 | 70 | — | — | 28.9 | 50.2 | 24.2 | — | — | — | — | — | — | — | — | |
| AVCapBackbone=finetuning2025.01 | 75.8 | 17.8 | 46.8 | 68.1 | — | — | 28.7 | 49.1 | 24.3 | — | — | — | — | — | — | — | — | |
| LOAE+2025.01 | 75.6 | 18.2 | 46.9 | 69.8 | — | — | 25.8 | 49.2 | 24.8 | — | — | — | — | — | — | — | — | |
| PairMixPairMix=true, Multi-TTA=false2022.10 | 75.5 | 17.7 | 46.6 | 69.3 | 52.9 | 38.9 | 28.3 | 49.9 | 24.1 | — | — | — | — | — | — | — | — | |
| BART-tagsPre-training Dataset=AS2025.01 | 75.3 | 17.6 | 46.5 | 69.9 | — | — | 26.6 | 49.3 | 24.1 | — | — | — | — | — | — | — | — | |
| BART-tagsPretraining Data=AudioSet (AS)2024.06 | 75.3 | 17.6 | 46.5 | 69.9 | — | — | 26.6 | 49.3 | 24.1 | — | — | — | — | — | — | — | — | |
| SLAP2026.01 | 75.1 | 18.1 | — | — | — | — | — | — | 24.9 | — | — | — | — | — | — | — | — | |
| AVCapBackbone=freeze2025.01 | 74.4 | 16.2 | 45.5 | 70.8 | — | — | 29.5 | 49.8 | 22.8 | — | — | — | — | — | — | — | — | |
| CNN-GPT22025.01 | 73.3 | 17.7 | 45.5 | 71.3 | — | — | 30.9 | 50.3 | 24 | — | — | — | — | — | — | — | — | |
| BLAT2026.01 | 73.3 | 18.4 | — | — | — | — | — | — | 24.7 | — | — | — | — | — | — | — | — | |
| Cacophony2026.01 | 72.8 | 16.8 | — | — | — | — | — | — | 23.6 | — | — | — | — | — | — | — | — | |
| M2D22026.01 | 72.4 | 17.6 | — | — | — | — | — | — | 24.3 | — | — | — | — | — | — | — | — | |
| AnyMALZeroshot=true, LLM Size=13B, Audio Encoder=CLAP2023.09 | 72.1 | 22 | 47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HTSAT-BARTPretraining Data=None2024.06 | 72.1 | 16.9 | 44.5 | 67.5 | — | — | 27.2 | 48.3 | 23.7 | — | — | — | — | — | — | — | — | |
| V-ACT2025.01 | 71.1 | 17.2 | 44.2 | 69.8 | — | — | 28.1 | 49.4 | 23.7 | — | — | — | — | — | — | — | — | |
| V-ACTPretraining Data=None2024.06 | 71.1 | 17.2 | 44.2 | 69.8 | — | — | 28.1 | 49.4 | 23.7 | — | — | — | — | — | — | — | — | |
| SLAM-LLMScenario=In Domain, Zero-shot=true2026.01 | 70.5 | 18 | 44.2 | — | — | — | — | — | 25.3 | 66.2 | — | — | — | — | — | — | — | |
| t→a PDPre-training=WavCaps+SoundVECaps, Beam size=52026.05 | 70.5 | 18.5 | 44.5 | — | — | — | 25.9 | 49.9 | 24.9 | — | — | — | — | — | — | — | — | |
| DRCapPre-training=WavCaps+SoundVECaps2026.05 | 70.5 | 18 | 44.2 | — | — | — | — | — | 25.3 | — | — | — | — | — | — | — | — | |
| AnyMALZeroshot=true, LLM Size=7B, Audio Encoder=CLAP2023.09 | 70.4 | 21 | 45.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| t100→a100Pre-training=WavCaps+SoundVECaps, Beam size=5, Embedding Dimension=1002026.05 | 69.2 | 17.5 | 43.3 | — | — | — | 25.1 | 49.7 | 24.6 | — | — | — | — | — | — | — | — | |
| ACTZeroshot=true2023.09 | 67.9 | 16 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ACTPre-training Dataset=AS2025.01 | 67.9 | 16 | 42 | 64.7 | — | — | 25.2 | 46.8 | 22.2 | — | — | — | — | — | — | — | — | |
| PANNS + BERTZeroshot=true2023.09 | 66.7 | 17.2 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| a→aPre-training=WavCaps+SoundVECaps, Beam size=52026.05 | 66.6 | 18.2 | 42.4 | — | — | — | 25.3 | 48.7 | 24.1 | — | — | — | — | — | — | — | — | |
| CNN10-VGGZeroshot=true2023.09 | 66 | 16.8 | 41.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| t→a PDPre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=52026.05 | 65.1 | 17.8 | 41.5 | — | — | — | 23 | 47.1 | 24.9 | — | — | — | — | — | — | — | — | |
| GAMAPretraining Data=None2024.06 | 64.8 | 5.4 | 35.1 | 20.4 | — | — | 12.2 | 19.1 | 16.3 | — | — | — | — | — | — | — | — | |
| Zhang et al.Scenario=In Domain, Zero-shot=true2026.01 | 64.4 | 15.6 | 40 | — | — | — | — | — | 22 | — | — | — | — | — | — | — | — | |
| SoftHardPre-training=WavCaps, Backbone=HTSAT-BERT-ZS2026.05 | 64.4 | 15.6 | 40 | — | — | — | 21.3 | 45.7 | 22 | — | — | — | — | — | — | — | — | |
| t100→a100Pre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=5, Embedding Dimension=1002026.05 | 64.1 | 17.1 | 40.6 | — | — | — | 21.9 | 46.7 | 24.2 | — | — | — | — | — | — | — | — | |
| BaselinePairMix=false, Multi-TTA=false2022.10 | 63.5 | 16.8 | 40.2 | 65.4 | 47.5 | 33.6 | 23.4 | 47.1 | 22.4 | — | — | — | — | — | — | — | — | |
| WSACScenario=In Domain, Zero-shot=true2026.01 | 63.3 | 17.3 | 40.3 | — | — | — | — | — | 24.1 | — | — | — | — | — | — | — | — | |
| a→aPre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=52026.05 | 63 | 17.7 | 40.3 | — | — | — | 23.7 | 47.3 | 23.8 | — | — | — | — | — | — | — | — | |
| a100→a100Pre-training=WavCaps+SoundVECaps, Beam size=5, Embedding Dimension=1002026.05 | 62.8 | 17.7 | 40.2 | — | — | — | 23.4 | 47.8 | 23.7 | — | — | — | — | — | — | — | — | |
| a100→a100Pre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=5, Embedding Dimension=1002026.05 | 60.7 | 17.1 | 38.9 | — | — | — | 21.7 | 46.8 | 23.7 | — | — | — | — | — | — | — | — | |
| Ours2024.10 | 59.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TopDown-AlignedAttZeroshot=true2023.09 | 59.3 | 14.4 | 36.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| t→a ESPre-training=WavCaps+SoundVECaps, Beam size=52026.05 | 58.6 | 16.3 | 37.5 | — | — | — | 21.8 | 45.8 | 22.8 | — | — | — | — | — | — | — | — | |
| X-InstructBLIP2024.10 | 58.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| t→a NNDPre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=52026.05 | 57.9 | 17.1 | 37.5 | — | — | — | 21 | 44.1 | 23.8 | — | — | — | — | — | — | — | — | |
| WSACPre-training=WavCaps, Backbone=HTSAT-BERT-ZS2026.05 | 56.4 | 16.3 | 36.3 | — | — | — | 17.1 | 43.5 | 23.2 | — | — | — | — | — | — | — | — | |
| t→a ADPre-training=WavCaps+SoundVECaps, Beam size=52026.05 | 51.1 | 16.2 | 33.7 | — | — | — | 20.4 | 43.4 | 23.4 | — | — | — | — | — | — | — | — | |
| t100rec→a100recPre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=52026.05 | 49.7 | 14.8 | 32.2 | — | — | — | 16.7 | 42.1 | 21.7 | — | — | — | — | — | — | — | — | |
| UIO2-XXLModel Category=Generalist, Images=1B*, Parameters=6.8B2024.12 | 48.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OmniFlowModel Category=Generalist, Images=30M, Parameters=3.4B2024.12 | 48 | — | — | — | — | — | — | — | — | — | 0.254 | — | — | — | — | — | — | |
| t→a ESPre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=52026.05 | 46.8 | 13.9 | 30.4 | — | — | — | 16 | 40.8 | 20.9 | — | — | — | — | — | — | — | — | |
| SLAM-LLMScenario=Cross Domain, Zero-shot=true2026.01 | 44.3 | 17 | 30.6 | — | — | — | — | — | 22.9 | 62.6 | — | — | — | — | — | — | — | |
| t→a ADPre-training=WavCaps, Backbone=HTSAT-BERT-ZS, Beam size=52026.05 | 34.4 | 12.2 | 23.3 | — | — | — | 14.6 | 36.2 | 18.5 | — | — | — | — | — | — | — | — | |
| Zhang et al.Scenario=Cross Domain, Zero-shot=true2026.01 | 33.7 | 12.4 | 23 | — | — | — | — | — | 18.2 | 52.1 | — | — | — | — | — | — | — | |
| ZerAuCapSetting=Zero-shot, LLM=OPT2023.11 | 28.1 | 8.6 | 18.3 | — | — | — | 6.8 | 33.1 | 12.3 | — | — | — | — | — | — | — | — | |
| ZerAuCapScenario=not specified, Zero-shot=true2026.01 | 28.1 | 8.6 | 18.3 | — | — | — | — | — | 12.3 | — | — | — | — | — | — | — | — | |
| ChatBridge2024.10 | 26.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WSACScenario=Cross Domain, Zero-shot=true2026.01 | 25.6 | 12 | 18.8 | — | — | — | — | — | 17.3 | — | — | — | — | — | — | — | — | |
| Shaharabany et al.Setting=Zero-shot2023.11 | 9.2 | — | — | — | — | — | 9.8 | 8.2 | 8.6 | — | — | — | — | — | — | — | — | |
| CoDiModel Category=Generalist, Images=400M, Parameters=4.3B, evaluated with official checkpoints=true2024.12 | 7.9 | — | — | — | — | — | — | — | — | — | 0.206 | — | — | — | — | — | — | |
| LoRA-MCLepsilon=0.05, Decoding=BS, Beam=52025.07 | 1.237 | 0.265 | 0.728 | — | — | — | 0.309 | 0.636 | 0.399 | — | — | — | — | — | 0.542 | 0.491 | 0.715 | |
| LoRA-MCLvariant=annealed, Decoding=BS, Beam=52025.07 | 1.211 | 0.268 | 0.716 | — | — | — | 0.315 | 0.636 | 0.398 | — | — | — | — | — | 0.575 | 0.423 | 0.714 | |
| LoRA-MCLepsilon=0.05, Decoding=BS, Beam=22025.07 | 1.209 | 0.26 | 0.714 | — | — | — | 0.298 | 0.631 | 0.394 | — | — | — | — | — | 0.544 | 0.464 | 0.712 | |
| LoRA-MCLvariant=annealed, Decoding=BS, Beam=22025.07 | 1.205 | 0.264 | 0.712 | — | — | — | 0.309 | 0.632 | 0.392 | — | — | — | — | — | 0.574 | 0.411 | 0.714 | |
| LoRA-MCLepsilon=0.0005, Decoding=BS, Beam=52025.07 | 1.19 | 0.263 | 0.706 | — | — | — | 0.306 | 0.631 | 0.392 | — | — | — | — | — | 0.579 | 0.41 | 0.712 | |
| LoRA-MCLepsilon=0.05, Decoding=BS, Beam=12025.07 | 1.181 | 0.253 | 0.695 | — | — | — | 0.273 | 0.622 | 0.382 | — | — | — | — | — | 0.551 | 0.427 | 0.712 | |
| LoRA-MoEDecoding=BS, Beam=102025.07 | 1.161 | 0.256 | 0.692 | — | — | — | 0.288 | 0.613 | 0.385 | — | — | — | — | — | 0.411 | 0.746 | 0.703 | |
| LoRA-MLErank=8, Decoding=BS, Beam=102025.07 | 1.157 | 0.256 | 0.689 | — | — | — | 0.286 | 0.61 | 0.385 | — | — | — | — | — | 0.41 | 0.746 | 0.704 | |
| LoRA-MoEDecoding=BS, Beam=252025.07 | 1.154 | 0.258 | 0.689 | — | — | — | 0.287 | 0.612 | 0.385 | — | — | — | — | — | 0.415 | 0.736 | 0.705 | |
| LoRA-MLErank=40, Decoding=BS, Beam=252025.07 | 1.152 | 0.26 | 0.688 | — | — | — | 0.289 | 0.611 | 0.384 | — | — | — | — | — | 0.415 | 0.735 | 0.706 | |
| LoRA-MCLepsilon=0.0005, Decoding=BS, Beam=22025.07 | 1.152 | 0.258 | 0.684 | — | — | — | 0.277 | 0.626 | 0.385 | — | — | — | — | — | 0.575 | 0.401 | 0.712 | |
| LoRA-MLErank=40, Decoding=BS, Beam=102025.07 | 1.151 | 0.255 | 0.686 | — | — | — | 0.298 | 0.61 | 0.382 | — | — | — | — | — | 0.407 | 0.747 | 0.704 | |
| LoRA-MCLvariant=annealed, Decoding=BS, Beam=12025.07 | 1.149 | 0.255 | 0.68 | — | — | — | 0.275 | 0.615 | 0.379 | — | — | — | — | — | 0.57 | 0.397 | 0.713 | |
| LoRA-MLErank=40, Decoding=BS, Beam=52025.07 | 1.144 | 0.251 | 0.681 | — | — | — | 0.28 | 0.606 | 0.382 | — | — | — | — | — | 0.392 | 0.773 | 0.701 | |
| LoRA-MLErank=8, Decoding=BS, Beam=252025.07 | 1.144 | 0.258 | 0.683 | — | — | — | 0.281 | 0.609 | 0.383 | — | — | — | — | — | 0.417 | 0.732 | 0.706 | |
| LoRA-MLErank=40, Decoding=DBS (λ = 0.8), Beam=252025.07 | 1.139 | 0.253 | 0.68 | — | — | — | 0.282 | 0.605 | 0.381 | — | — | — | — | — | 0.399 | 0.763 | 0.702 | |
| LoRA-MLErank=40, Decoding=DBS (λ = 1.0), Beam=52025.07 | 1.138 | 0.25 | 0.672 | — | — | — | 0.268 | 0.614 | 0.378 | — | — | — | — | — | 0.58 | 0.403 | 0.709 |