# Natural Language Understanding on ITALIC original Source: [Natural Language Understanding on ITALIC original]() Collection index: [llms.txt]() Data updated: 2026-05-11T10:11:09.598Z Evaluation of model performance on the ITALIC original dataset for general Natural Language Understanding tasks, potentially focusing on specific evaluation modes like fast inference or Chain-of-Thought (COT) performance. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Natural Language Understanding | | Dataset (including any stated split) | ITALIC original | | Metric | Fast Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Fast Accuracy](): Higher is better; unit: PERCENT. - [COT Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Fast Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Ministral-3-8B-Instruct-2512-BF16 | 87.4 | Not recorded | 6a0162badbc836b87d82332a | | GPT-5 nano | 86.8 | Not recorded | 6a0162badbc836b87d82332a | | gemma-3-12b-it | 76.5 | Not recorded | 6a0162badbc836b87d82332a | | Qwen3-8B | 70.8 | Not recorded | 6a0162badbc836b87d82332a | | Llama-3.1-8B-Instruct | 70.7 | Not recorded | 6a0162badbc836b87d82332a | | Velvet-14B | 68.1 | Not recorded | 6a0162badbc836b87d82332a | | Qwen3-4B | 65.4 | Not recorded | 6a0162badbc836b87d82332a | | gemma-3-4b-it | 62.5 | Not recorded | 6a0162badbc836b87d82332a | | Moonlight-16B-A3B-Instruct | 61.7 | Not recorded | 6a0162badbc836b87d82332a | | EngGPT2-16B-A3B | 59.3 | Not recorded | 6a0162badbc836b87d82332a | | FastwebMIIA-7B | 59.3 | Not recorded | 6a0162badbc836b87d82332a | | Llama-3.2-3B-Instruct | 58.4 | Not recorded | 6a0162badbc836b87d82332a | | deepseek-moe-16b-chat | 50.9 | Not recorded | 6a0162badbc836b87d82332a | | Minerva-7B-instruct-v1.0 | 49.9 | Not recorded | 6a0162badbc836b87d82332a | | LLaMAntino-3-ANITA-8B-Inst-DPO-ITA | 44.8 | Not recorded | 6a0162badbc836b87d82332a | | gpt-oss-20b | 26.1 | Not recorded | 6a0162badbc836b87d82332a | ### Ministral-3-8B-Instruct-2512-BF16 Fast Accuracy: 87.4. Result ID: 6a01ab3c47f34ce924e41e90. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### GPT-5 nano Fast Accuracy: 86.8. Result ID: 6a01ab3c47f34ce924e41e91. [Reporting paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### gemma-3-12b-it Fast Accuracy: 76.5. Result ID: 6a01ab3c47f34ce924e41e8f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Qwen3-8B Fast Accuracy: 70.8. Result ID: 6a01ab3c47f34ce924e41e8b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Llama-3.1-8B-Instruct Fast Accuracy: 70.7. Result ID: 6a01ab3c47f34ce924e41e8d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Velvet-14B Fast Accuracy: 68.1. Result ID: 6a01ab3c47f34ce924e41e88. [Reporting paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Qwen3-4B Fast Accuracy: 65.4. Result ID: 6a01ab3c47f34ce924e41e8a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### gemma-3-4b-it Fast Accuracy: 62.5. Result ID: 6a01ab3c47f34ce924e41e8e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Moonlight-16B-A3B-Instruct Fast Accuracy: 61.7. Result ID: 6a01ab3c47f34ce924e41e94. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### EngGPT2-16B-A3B Fast Accuracy: 59.3. Result ID: 6a01ab3c47f34ce924e41e85. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### FastwebMIIA-7B Fast Accuracy: 59.3. Result ID: 6a01ab3c47f34ce924e41e86. [Reporting paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Llama-3.2-3B-Instruct Fast Accuracy: 58.4. Result ID: 6a01ab3c47f34ce924e41e8c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### deepseek-moe-16b-chat Fast Accuracy: 50.9. Result ID: 6a01ab3c47f34ce924e41e92. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### Minerva-7B-instruct-v1.0 Fast Accuracy: 49.9. Result ID: 6a01ab3c47f34ce924e41e87. [Reporting paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### LLaMAntino-3-ANITA-8B-Inst-DPO-ITA Fast Accuracy: 44.8. Result ID: 6a01ab3c47f34ce924e41e89. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. ### gpt-oss-20b Fast Accuracy: 26.1. Result ID: 6a01ab3c47f34ce924e41e93. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"evaluation\_type":"chat-based"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.