# Chime-in Reason on MUIR (test) Source: [Chime-in Reason on MUIR (test)]() Collection index: [llms.txt]() Data updated: 2026-03-04T13:07:02.811Z Evaluation of model performance on the Chime-in Reason task using the test split of the MUIR dataset, measured by Accuracy, Macro-F1, and Weighted scores. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Chime-in Reason | | Dataset (including any stated split) | MUIR (test) | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 17 of 17 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [Macro F1 Score](): Higher is better; unit: PERCENT. - [Weighted Score](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Human Evaluator | 88.59 | Not recorded | 69a66c132d603e6de3e3cd80 | | GPT-4o | 87.16 | Not recorded | 69a66c132d603e6de3e3cd80 | | Qwen-2.5-Instruct | 86.28 | Not recorded | 69a66c132d603e6de3e3cd80 | | Qwen3-Max | 83.33 | Not recorded | 69a66c132d603e6de3e3cd80 | | Gemini-2.5-Pro | 83.27 | Not recorded | 69a66c132d603e6de3e3cd80 | | Llama-3.1-Instruct | 82.84 | Not recorded | 69a66c132d603e6de3e3cd80 | | Qwen-3 | 81.34 | Not recorded | 69a66c132d603e6de3e3cd80 | | DeepSeek-V3.2 | 81.22 | Not recorded | 69a66c132d603e6de3e3cd80 | | Llama-3.2-Instruct | 80.95 | Not recorded | 69a66c132d603e6de3e3cd80 | | Qwen-2.5-Instruct | 80.69 | Not recorded | 69a66c132d603e6de3e3cd80 | | Phi-4-Mini-Instruct | 79.85 | Not recorded | 69a66c132d603e6de3e3cd80 | | Gemma-2-it | 79.41 | Not recorded | 69a66c132d603e6de3e3cd80 | | Qwen-3 | 78.59 | Not recorded | 69a66c132d603e6de3e3cd80 | | Gte-large-en-v1.5 | 35.6 | Not recorded | 69a66c132d603e6de3e3cd80 | | Jina-embedding-v3 | 30.45 | Not recorded | 69a66c132d603e6de3e3cd80 | | Bge-m3 | 28.6 | Not recorded | 69a66c132d603e6de3e3cd80 | | Random Guess | 17.21 | Not recorded | 69a66c132d603e6de3e3cd80 | ### Human Evaluator Accuracy: 88.59. Result ID: 69a82e5f6dbb7075b94e8d21. [Reporting paper]() Recorded evaluation configuration: {"Method":"Baseline","Size":"N/A"}. This record does not assert independent reproduction. ### GPT-4o Accuracy: 87.16. Result ID: 69a82e5f6dbb7075b94e8d25. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM + Prompt","Size":"N/A"}. This record does not assert independent reproduction. ### Qwen-2.5-Instruct Accuracy: 86.28. Result ID: 69a82e5f6dbb7075b94e8d2a. [Reporting paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"3B"}. This record does not assert independent reproduction. ### Qwen3-Max Accuracy: 83.33. Result ID: 69a82e5f6dbb7075b94e8d22. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"LLM + Prompt","Size":"N/A"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro Accuracy: 83.27. Result ID: 69a82e5f6dbb7075b94e8d23. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM + Prompt","Size":"N/A"}. This record does not assert independent reproduction. ### Llama-3.1-Instruct Accuracy: 82.84. Result ID: 69a82e5f6dbb7075b94e8d2f. [Reporting paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"8B"}. This record does not assert independent reproduction. ### Qwen-3 Accuracy: 81.34. Result ID: 69a82e5f6dbb7075b94e8d30. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"8B"}. This record does not assert independent reproduction. ### DeepSeek-V3.2 Accuracy: 81.22. Result ID: 69a82e5f6dbb7075b94e8d24. [Reporting paper]() Recorded evaluation configuration: {"Method":"LLM + Prompt","Size":"N/A"}. This record does not assert independent reproduction. ### Llama-3.2-Instruct Accuracy: 80.95. Result ID: 69a82e5f6dbb7075b94e8d2b. [Reporting paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"3B"}. This record does not assert independent reproduction. ### Qwen-2.5-Instruct Accuracy: 80.69. Result ID: 69a82e5f6dbb7075b94e8d2e. [Reporting paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"7B"}. This record does not assert independent reproduction. ### Phi-4-Mini-Instruct Accuracy: 79.85. Result ID: 69a82e5f6dbb7075b94e8d2c. [Reporting paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"3.8B"}. This record does not assert independent reproduction. ### Gemma-2-it Accuracy: 79.41. Result ID: 69a82e5f6dbb7075b94e8d29. [Reporting paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"2B"}. This record does not assert independent reproduction. ### Qwen-3 Accuracy: 78.59. Result ID: 69a82e5f6dbb7075b94e8d2d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method":"SLM + Fine-Tuning","Size":"4B"}. This record does not assert independent reproduction. ### Gte-large-en-v1.5 Accuracy: 35.6. Result ID: 69a82e5f6dbb7075b94e8d26. [Reporting paper]() Recorded evaluation configuration: {"Method":"Embedding Model + KNN","Size":"0.4B"}. This record does not assert independent reproduction. ### Jina-embedding-v3 Accuracy: 30.45. Result ID: 69a82e5f6dbb7075b94e8d28. [Reporting paper]() Recorded evaluation configuration: {"Method":"Embedding Model + KNN","Size":"0.6B"}. This record does not assert independent reproduction. ### Bge-m3 Accuracy: 28.6. Result ID: 69a82e5f6dbb7075b94e8d27. [Reporting paper]() Recorded evaluation configuration: {"Method":"Embedding Model + KNN","Size":"0.5B"}. This record does not assert independent reproduction. ### Random Guess Accuracy: 17.21. Result ID: 69a82e5f6dbb7075b94e8d20. [Reporting paper]() Recorded evaluation configuration: {"Method":"Baseline","Size":"N/A"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.