# Zero-shot Language Understanding on ARC-C, BoolQ, Lambada, PIQA, and Winogrande Source: [Zero-shot Language Understanding on ARC-C, BoolQ, Lambada, PIQA, and Winogrande]() Collection index: [llms.txt]() Data updated: 2026-03-31T09:10:41.432Z Evaluation of zero-shot performance across the specific suite of language understanding tasks: ARC-Challenge, BoolQ, Lambada, PIQA, and Winogrande. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Language Understanding | | Dataset (including any stated split) | Benchmarks (ARC-C, BoolQ, Lambada, PIQA, Winogrande) Zero-shot | | Metric | ARC-C Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [ARC-C Accuracy](): Higher is better; unit: PERCENT. - [BoolQ Accuracy](): Higher is better; unit: PERCENT. - [Lambada Accuracy](): Higher is better; unit: PERCENT. - [PIQA Accuracy](): Higher is better; unit: PERCENT. - [Winogrande Accuracy](): Higher is better; unit: PERCENT. - [Average Score](): Higher is better; unit: PERCENT. ## Recorded results | Method | ARC-C Accuracy | Result date | Report paper | | --- | --- | --- | --- | | MicroMix | 56.66 | Not recorded | 68fa7a6a1e209131cea83140 | | FlatQuant | 56.23 | Not recorded | 68fa7a6a1e209131cea83140 | | FP16 | 55.89 | Not recorded | 68fa7a6a1e209131cea83140 | | INT6 | 55.29 | Not recorded | 68fa7a6a1e209131cea83140 | | Atom | 54.78 | Not recorded | 68fa7a6a1e209131cea83140 | | FP16 | 53.58 | Not recorded | 68fa7a6a1e209131cea83140 | | QuaRot | 53.08 | Not recorded | 68fa7a6a1e209131cea83140 | | QUIK | 52.9 | Not recorded | 68fa7a6a1e209131cea83140 | | FlatQuant | 51.54 | Not recorded | 68fa7a6a1e209131cea83140 | | AMXFP4 | 51.54 | Not recorded | 68fa7a6a1e209131cea83140 | | MicroMix | 50.26 | Not recorded | 68fa7a6a1e209131cea83140 | | Atom | 50.17 | Not recorded | 68fa7a6a1e209131cea83140 | | INT6 | 48.38 | Not recorded | 68fa7a6a1e209131cea83140 | | QuaRot | 46.42 | Not recorded | 68fa7a6a1e209131cea83140 | | QUIK | 44.62 | Not recorded | 68fa7a6a1e209131cea83140 | | AMXFP4 | 43.77 | Not recorded | 68fa7a6a1e209131cea83140 | ### MicroMix ARC-C Accuracy: 56.66. Result ID: 69cb8e9347f34ce924b410d5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"5.22","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### FlatQuant ARC-C Accuracy: 56.23. Result ID: 69cb8e9347f34ce924b410d2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"4.71","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### FP16 ARC-C Accuracy: 55.89. Result ID: 69cb8e9347f34ce924b410ce. [Reporting paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"16.00","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### INT6 ARC-C Accuracy: 55.29. Result ID: 69cb8e9347f34ce924b410d4. [Reporting paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"6.00","Evaluation Protocol":"0-shot","Quantization Implementation":"symmetric per-token"}. This record does not assert independent reproduction. ### Atom ARC-C Accuracy: 54.78. Result ID: 69cb8e9347f34ce924b410d1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"4.21","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### FP16 ARC-C Accuracy: 53.58. Result ID: 69cb8e9347f34ce924b410c6. [Reporting paper]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"16.00","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### QuaRot ARC-C Accuracy: 53.08. Result ID: 69cb8e9347f34ce924b410cf. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"4.12","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### QUIK ARC-C Accuracy: 52.9. Result ID: 69cb8e9347f34ce924b410d0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"6.21","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### FlatQuant ARC-C Accuracy: 51.54. Result ID: 69cb8e9347f34ce924b410ca. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"4.19","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### AMXFP4 ARC-C Accuracy: 51.54. Result ID: 69cb8e9347f34ce924b410d3. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Qwen2.5-32B","Avg. Bits":"5.00","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### MicroMix ARC-C Accuracy: 50.26. Result ID: 69cb8e9347f34ce924b410cd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"5.51","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### Atom ARC-C Accuracy: 50.17. Result ID: 69cb8e9347f34ce924b410c9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"4.25","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### INT6 ARC-C Accuracy: 48.38. Result ID: 69cb8e9347f34ce924b410cc. [Reporting paper]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"6.00","Evaluation Protocol":"0-shot","Quantization Implementation":"symmetric per-token"}. This record does not assert independent reproduction. ### QuaRot ARC-C Accuracy: 46.42. Result ID: 69cb8e9347f34ce924b410c7. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"4.12","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### QUIK ARC-C Accuracy: 44.62. Result ID: 69cb8e9347f34ce924b410c8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"5.95","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. ### AMXFP4 ARC-C Accuracy: 43.77. Result ID: 69cb8e9347f34ce924b410cb. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama3.1-8B","Avg. Bits":"5.00","Evaluation Protocol":"0-shot"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.