# Multimodal Audio Understanding on MMAU mini v05.15.25 (test) Source: [Multimodal Audio Understanding on MMAU mini v05.15.25 (test)]() Collection index: [llms.txt]() Data updated: 2026-03-11T09:04:00.289Z Evaluation of Multimodal Audio Understanding capabilities using the test split of the MMAU mini dataset (version 05.15.25), reporting scores across Sound, Music, and Speech domains. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Multimodal Audio Understanding | | Dataset (including any stated split) | MMAU mini v05.15.25 (test) | | Metric | Sound Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 25 of 25 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Sound Accuracy](): Higher is better; unit: PERCENT. - [Music Accuracy](): Higher is better; unit: PERCENT. - [Speech Accuracy](): Higher is better; unit: PERCENT. - [Overall Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Sound Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Audio Flamingo 3 | 79.6 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Qwen2.5-Omni | 78.1 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Audio Flamingo 2 | 71.5 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | DeSTA-2.5-Audio | 70.3 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Audio Reasoner | 67.9 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Qwen2-Audio-Instruct | 67.3 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | ALARM-CA | 66.4 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Phi-4-Multimodal | 65.5 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | GPT-4o Audio | 64.6 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | ALARM-E | 64 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | AL-SSLAM-R | 61.3 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | ALARM-P | 58.6 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Gemma 3n | 55.9 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | AL-Whisper-R | 52.3 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | Gemma 3n | 51.4 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | GPT-4o mini Audio | 50.8 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | AL-MuQ-R | 50.5 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | AL-W2VBERT2-R | 46.2 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | AL-Whisper-Instruct-R | 45.9 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | MusiLingo | 43.2 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | SALMONN | 41.1 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | MU-LLaMA | 33 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | GAMA | 31.8 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | GAMA-IT | 30.9 | Not recorded | 69b0f72f2d603e6de3e45fb9 | | LTU | 20.4 | Not recorded | 69b0f72f2d603e6de3e45fb9 | ### Audio Flamingo 3 Sound Accuracy: 79.6. Result ID: 69b12ff96dbb7075b95bc71d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8B","No LLM Modification":"×","# Samples":"26.7M","# Hours":"54.4K"}. This record does not assert independent reproduction. ### Qwen2.5-Omni Sound Accuracy: 78.1. Result ID: 69b12ff96dbb7075b95bc71c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"×","# Tokens":"0.3T"}. This record does not assert independent reproduction. ### Audio Flamingo 2 Sound Accuracy: 71.5. Result ID: 69b12ff96dbb7075b95bc71f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"3B","No LLM Modification":"×","# Samples":"5.9M"}. This record does not assert independent reproduction. ### DeSTA-2.5-Audio Sound Accuracy: 70.3. Result ID: 69b12ff96dbb7075b95bc71a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8B","No LLM Modification":"✓","# Samples":"5M","# Hours":"7K"}. This record does not assert independent reproduction. ### Audio Reasoner Sound Accuracy: 67.9. Result ID: 69b12ff96dbb7075b95bc71b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"8B","No LLM Modification":"×","# Samples":"\> 1.2M","# Hours":"\> 320K"}. This record does not assert independent reproduction. ### Qwen2-Audio-Instruct Sound Accuracy: 67.3. Result ID: 69b12ff96dbb7075b95bc719. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"×","# Hours":"320K"}. This record does not assert independent reproduction. ### ALARM-CA Sound Accuracy: 66.4. Result ID: 69b12ff96dbb7075b95bc725. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"5.5M","# Hours":"17K","# Tokens":"1.5B"}. This record does not assert independent reproduction. ### Phi-4-Multimodal Sound Accuracy: 65.5. Result ID: 69b12ff96dbb7075b95bc71e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"×","# Samples":"\> 28M","# Hours":"\> 2M"}. This record does not assert independent reproduction. ### GPT-4o Audio Sound Accuracy: 64.6. Result ID: 69b12ff96dbb7075b95bc710. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### ALARM-E Sound Accuracy: 64. Result ID: 69b12ff96dbb7075b95bc727. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"5.5M","# Hours":"17K","# Tokens":"1.5B"}. This record does not assert independent reproduction. ### AL-SSLAM-R Sound Accuracy: 61.3. Result ID: 69b12ff96dbb7075b95bc723. [Reporting paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"0.4M","# Hours":"1.2K","# Tokens":"108M"}. This record does not assert independent reproduction. ### ALARM-P Sound Accuracy: 58.6. Result ID: 69b12ff96dbb7075b95bc726. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"5.5M","# Hours":"17K","# Tokens":"1.5B"}. This record does not assert independent reproduction. ### Gemma 3n Sound Accuracy: 55.9. Result ID: 69b12ff96dbb7075b95bc718. [Reporting paper]() Recorded evaluation configuration: {"Size":"8B4E"}. This record does not assert independent reproduction. ### AL-Whisper-R Sound Accuracy: 52.3. Result ID: 69b12ff96dbb7075b95bc724. [Reporting paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"5.5M","# Hours":"17K","# Tokens":"1.5B"}. This record does not assert independent reproduction. ### Gemma 3n Sound Accuracy: 51.4. Result ID: 69b12ff96dbb7075b95bc717. [Reporting paper]() Recorded evaluation configuration: {"Size":"6B2E"}. This record does not assert independent reproduction. ### GPT-4o mini Audio Sound Accuracy: 50.8. Result ID: 69b12ff96dbb7075b95bc70f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### AL-MuQ-R Sound Accuracy: 50.5. Result ID: 69b12ff96dbb7075b95bc722. [Reporting paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"0.4M","# Hours":"1.5K","# Tokens":"137M"}. This record does not assert independent reproduction. ### AL-W2VBERT2-R Sound Accuracy: 46.2. Result ID: 69b12ff96dbb7075b95bc721. [Reporting paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"0.5M","# Hours":"0.6K","# Tokens":"58M"}. This record does not assert independent reproduction. ### AL-Whisper-Instruct-R Sound Accuracy: 45.9. Result ID: 69b12ff96dbb7075b95bc720. [Reporting paper]() Recorded evaluation configuration: {"Size":"4B","No LLM Modification":"✓","# Samples":"0.5M","# Hours":"1K","# Tokens":"91M"}. This record does not assert independent reproduction. ### MusiLingo Sound Accuracy: 43.2. Result ID: 69b12ff96dbb7075b95bc716. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"✓","# Samples":"0.7M"}. This record does not assert independent reproduction. ### SALMONN Sound Accuracy: 41.1. Result ID: 69b12ff96dbb7075b95bc715. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"13B","No LLM Modification":"×","# Samples":"2.3M","# Hours":"4.4K"}. This record does not assert independent reproduction. ### MU-LLaMA Sound Accuracy: 33. Result ID: 69b12ff96dbb7075b95bc714. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"✓","# Samples":"0.1M","# Hours":"76"}. This record does not assert independent reproduction. ### GAMA Sound Accuracy: 31.8. Result ID: 69b12ff96dbb7075b95bc712. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"×","# Samples":"6.3M"}. This record does not assert independent reproduction. ### GAMA-IT Sound Accuracy: 30.9. Result ID: 69b12ff96dbb7075b95bc713. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"×","# Samples":"6.5M"}. This record does not assert independent reproduction. ### LTU Sound Accuracy: 20.4. Result ID: 69b12ff96dbb7075b95bc711. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Size":"7B","No LLM Modification":"×","# Samples":"5.6M"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.