# Zero-shot Language Understanding Suite (LMB, HellA, PIQA, ARC-E/C, WINO, Open, MMLU)

Source: [Zero-shot Language Understanding Suite (LMB, HellA, PIQA, ARC-E/C, WINO, Open, MMLU)](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:23:38.917Z

Evaluation of zero-shot performance across a custom suite of language understanding benchmarks including LMB, HellaSwag, PIQA, ARC (Easy/Challenge), WinoGrande, Open, and MMLU.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Zero-shot Language Understanding |
| Dataset (including any stated split) | Evaluation Suite Zero-shot (LMB, HellA, PIQA, ARC-E, ARC-C, WINO, Open, MMLU) |
| Metric | ARC-E Accuracy |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 25 of 25 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [ARC-E Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=ARC-E%20Accuracy>): Higher is better; unit: PERCENT.

- [LMB Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=LMB%20Accuracy>): Higher is better; unit: PERCENT.

- [HellA Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=HellA%20Accuracy>): Higher is better; unit: PERCENT.

- [PIQA Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=PIQA%20Accuracy>): Higher is better; unit: PERCENT.

- [ARC-C Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=ARC-C%20Accuracy>): Higher is better; unit: PERCENT.

- [WinoGrande Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=WinoGrande%20Accuracy>): Higher is better; unit: PERCENT.

- [Open Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=Open%20Accuracy>): Higher is better; unit: PERCENT.

- [MMLU Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=MMLU%20Accuracy>): Higher is better; unit: PERCENT.

- [Average Drop](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=Average%20Drop>): Lower is better; unit: PERCENT.

- [BoolQ Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=BoolQ%20Accuracy>): Higher is better; unit: PERCENT.

- [SciQ Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=SciQ%20Accuracy>): Higher is better; unit: PERCENT.

- [Average Accuracy](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=Average%20Accuracy>): Higher is better; unit: PERCENT.

- [Performance Delta](<https://www.sota2.com/research/sota/zero-shot-language-understanding-suite-lmb-hella-piqa-arc-e-c-wino-open-mmlu.md?metric=Performance%20Delta>): Higher is better; unit: SCALAR.

## Recorded results

| Method | ARC-E Accuracy | Result date | Report paper |
| --- | --- | --- | --- |
| BLT Distill (Teacher) | 83.4 | Not recorded | 6981812fa8b01e76cc8e231f |
| Ours (Qwen 3 4B Teacher) | 78.9 | Not recorded | 6981812fa8b01e76cc8e231f |
| Ours (Stage 2) | 77.1 | Not recorded | 6981812fa8b01e76cc8e231f |
| ALM (Teacher) | 76.9 | Not recorded | 6981812fa8b01e76cc8e231f |
| Ours (Stage 1) | 76.8 | Not recorded | 6981812fa8b01e76cc8e231f |
| Ours (Llama 3.2 3B Teacher) | 74.6 | Not recorded | 6981812fa8b01e76cc8e231f |
| H-Net Distill (Teacher) | 74.5 | Not recorded | 6981812fa8b01e76cc8e231f |
| ALM (Student) | 73.7 | Not recorded | 6981812fa8b01e76cc8e231f |
| Ours (Stage 1) | 73.6 | Not recorded | 6981812fa8b01e76cc8e231f |
| H-Net Distill (Student) | 72.1 | Not recorded | 6981812fa8b01e76cc8e231f |
| StableQAT | 72.05 | Not recorded | 6979980ea2b28e761d87cbc0 |
| ParetoQ | 71.83 | Not recorded | 6979980ea2b28e761d87cbc0 |
| Baseline | 71.63 | Not recorded | 6979980ea2b28e761d87cbc0 |
| ParetoQ | 71.47 | Not recorded | 6979980ea2b28e761d87cbc0 |
| StableQAT | 71.04 | Not recorded | 6979980ea2b28e761d87cbc0 |
| DSQ | 70.19 | Not recorded | 6979980ea2b28e761d87cbc0 |
| Ours (Stage 2) | 69.5 | Not recorded | 6981812fa8b01e76cc8e231f |
| DSQ | 69.19 | Not recorded | 6979980ea2b28e761d87cbc0 |
| StableQAT | 68.48 | Not recorded | 6979980ea2b28e761d87cbc0 |
| BLT Distill (Student) | 66.6 | Not recorded | 6981812fa8b01e76cc8e231f |
| DSQ | 66.58 | Not recorded | 6979980ea2b28e761d87cbc0 |
| ParetoQ | 65.73 | Not recorded | 6979980ea2b28e761d87cbc0 |
| Baseline no QAT | 61.99 | Not recorded | 6979980ea2b28e761d87cbc0 |
| Baseline no QAT | 26.14 | Not recorded | 6979980ea2b28e761d87cbc0 |
| Baseline no QAT | 25.55 | Not recorded | 6979980ea2b28e761d87cbc0 |

### BLT Distill (Teacher)

ARC-E Accuracy: 83.4. Result ID: 6996588f5698665ec49df04a.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 2 7B","Role":"Teacher"}. This record does not assert independent reproduction.

### Ours (Qwen 3 4B Teacher)

ARC-E Accuracy: 78.9. Result ID: 6996588f5698665ec49df053.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Qwen 3 4B","Role":"Teacher (Baseline)"}. This record does not assert independent reproduction.

### Ours (Stage 2)

ARC-E Accuracy: 77.1. Result ID: 6996588f5698665ec49df055.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Qwen 3 4B","Role":"Student","Bytes":"95B"}. This record does not assert independent reproduction.

### ALM (Teacher)

ARC-E Accuracy: 76.9. Result ID: 6996588f5698665ec49df04c.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Teacher"}. This record does not assert independent reproduction.

### Ours (Stage 1)

ARC-E Accuracy: 76.8. Result ID: 6996588f5698665ec49df054.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Qwen 3 4B","Role":"Student","Bytes":"30B"}. This record does not assert independent reproduction.

### Ours (Llama 3.2 3B Teacher)

ARC-E Accuracy: 74.6. Result ID: 6996588f5698665ec49df050.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Teacher (Baseline)"}. This record does not assert independent reproduction.

### H-Net Distill (Teacher)

ARC-E Accuracy: 74.5. Result ID: 6996588f5698665ec49df04e.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Teacher"}. This record does not assert independent reproduction.

### ALM (Student)

ARC-E Accuracy: 73.7. Result ID: 6996588f5698665ec49df04d.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Student (Byte)"}. This record does not assert independent reproduction.

### Ours (Stage 1)

ARC-E Accuracy: 73.6. Result ID: 6996588f5698665ec49df051.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Student","Bytes":"30B"}. This record does not assert independent reproduction.

### H-Net Distill (Student)

ARC-E Accuracy: 72.1. Result ID: 6996588f5698665ec49df04f.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Student (Byte)","Bytes":"189B"}. This record does not assert independent reproduction.

### StableQAT

ARC-E Accuracy: 72.05. Result ID: 699910385698665ec47613bd.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"4","Setting":"20B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### ParetoQ

ARC-E Accuracy: 71.83. Result ID: 699910385698665ec47613bb.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7c231e209131cea8ca56.md>)

Recorded evaluation configuration: {"Bits":"4","Setting":"20B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### Baseline

ARC-E Accuracy: 71.63. Result ID: 699910385698665ec47613b9.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"16","Setting":"Baseline"}. This record does not assert independent reproduction.

### ParetoQ

ARC-E Accuracy: 71.47. Result ID: 699910385698665ec47613bf.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7c231e209131cea8ca56.md>)

Recorded evaluation configuration: {"Bits":"3","Setting":"20B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### StableQAT

ARC-E Accuracy: 71.04. Result ID: 699910385698665ec47613c1.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"3","Setting":"20B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### DSQ

ARC-E Accuracy: 70.19. Result ID: 699910385698665ec47613bc.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa663c1e209131cea0f140.md>)

Recorded evaluation configuration: {"Bits":"4","Setting":"20B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### Ours (Stage 2)

ARC-E Accuracy: 69.5. Result ID: 6996588f5698665ec49df052.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 3.2 3B","Role":"Student","Bytes":"95B"}. This record does not assert independent reproduction.

### DSQ

ARC-E Accuracy: 69.19. Result ID: 699910385698665ec47613c4.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa663c1e209131cea0f140.md>)

Recorded evaluation configuration: {"Bits":"2","Setting":"30B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### StableQAT

ARC-E Accuracy: 68.48. Result ID: 699910385698665ec47613c5.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"2","Setting":"30B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### BLT Distill (Student)

ARC-E Accuracy: 66.6. Result ID: 6996588f5698665ec49df04b.

[Reporting paper](<https://www.sota2.com/research/paper/6981812fa8b01e76cc8e231f.md>)

Recorded evaluation configuration: {"Model":"Llama 2 7B","Role":"Student (Byte)","Bytes":"220B"}. This record does not assert independent reproduction.

### DSQ

ARC-E Accuracy: 66.58. Result ID: 699910385698665ec47613c0.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa663c1e209131cea0f140.md>)

Recorded evaluation configuration: {"Bits":"3","Setting":"20B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### ParetoQ

ARC-E Accuracy: 65.73. Result ID: 699910385698665ec47613c3.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7c231e209131cea8ca56.md>)

Recorded evaluation configuration: {"Bits":"2","Setting":"30B Tokens & lr=1e-4"}. This record does not assert independent reproduction.

### Baseline no QAT

ARC-E Accuracy: 61.99. Result ID: 699910385698665ec47613ba.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"4","Setting":"Baseline no QAT"}. This record does not assert independent reproduction.

### Baseline no QAT

ARC-E Accuracy: 26.14. Result ID: 699910385698665ec47613be.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"3","Setting":"Baseline no QAT"}. This record does not assert independent reproduction.

### Baseline no QAT

ARC-E Accuracy: 25.55. Result ID: 699910385698665ec47613c2.

[Reporting paper](<https://www.sota2.com/research/paper/6979980ea2b28e761d87cbc0.md>)

Recorded evaluation configuration: {"Bits":"2","Setting":"Baseline no QAT"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
