# Zero-shot Evaluation on ARC, BoolQ, HellaSwag, LAMBADA, and MMLU

Source: [Zero-shot Evaluation on ARC, BoolQ, HellaSwag, LAMBADA, and MMLU](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:23:38.277Z

Zero-shot performance evaluation across a suite of five key NLP downstream tasks: ARC (Challenge/Easy), BoolQ, HellaSwag, LAMBADA, and MMLU.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Zero-shot NLP Evaluation |
| Dataset (including any stated split) | NLP Downstream Benchmarks (ARC, BoolQ, HellaSwag, LAMBADA, MMLU) |
| Metric | ARC-C |
| Unit | SCALAR |
| Direction | Higher is better |
| Results loaded | 15 of 15 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [ARC-C](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=ARC-C>): Higher is better; unit: SCALAR.

- [ARC-E](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=ARC-E>): Higher is better; unit: SCALAR.

- [BoolQ](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=BoolQ>): Higher is better; unit: SCALAR.

- [HellaSwag](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=HellaSwag>): Higher is better; unit: SCALAR.

- [LAMBADA](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=LAMBADA>): Higher is better; unit: SCALAR.

- [MMLU](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=MMLU>): Higher is better; unit: SCALAR.

- [Avg. Accuracy (6 tasks)](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-arc-boolq-hellaswag-lambada-and-mmlu.md?metric=Avg.%20Accuracy%20(6%20tasks)>): Higher is better; unit: SCALAR.

## Recorded results

| Method | ARC-C | Result date | Report paper |
| --- | --- | --- | --- |
| Muon-32 | 0.354 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-32 | 0.352 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-8D | 0.347 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-32 | 0.298 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-32 | 0.296 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-8D | 0.293 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-32 | 0.281 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-32 | 0.278 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-8D | 0.272 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-32 | 0.247 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-32 | 0.236 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-8D | 0.235 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-32 | 0.211 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-8L/AdamW-8D | 0.205 | Not recorded | 68fa7bae1e209131cea89d3b |
| Muon-32 | 0.204 | Not recorded | 68fa7bae1e209131cea89d3b |

### Muon-32

ARC-C: 0.354. Result ID: 699a8d1a5698665ec41765b7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea70fcd.md>)

Recorded evaluation configuration: {"Model size":"XXL"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-32

ARC-C: 0.352. Result ID: 699a8d1a5698665ec41765b8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"XXL"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-8D

ARC-C: 0.347. Result ID: 699a8d1a5698665ec41765b9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"XXL"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-32

ARC-C: 0.298. Result ID: 699a8d1a5698665ec41765b5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"XL"}. This record does not assert independent reproduction.

### Muon-32

ARC-C: 0.296. Result ID: 699a8d1a5698665ec41765b4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea70fcd.md>)

Recorded evaluation configuration: {"Model size":"XL"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-8D

ARC-C: 0.293. Result ID: 699a8d1a5698665ec41765b6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"XL"}. This record does not assert independent reproduction.

### Muon-32

ARC-C: 0.281. Result ID: 699a8d1a5698665ec41765b1.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea70fcd.md>)

Recorded evaluation configuration: {"Model size":"Large"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-32

ARC-C: 0.278. Result ID: 699a8d1a5698665ec41765b2.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"Large"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-8D

ARC-C: 0.272. Result ID: 699a8d1a5698665ec41765b3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"Large"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-32

ARC-C: 0.247. Result ID: 699a8d1a5698665ec41765af.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"Medium"}. This record does not assert independent reproduction.

### Muon-32

ARC-C: 0.236. Result ID: 699a8d1a5698665ec41765ae.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea70fcd.md>)

Recorded evaluation configuration: {"Model size":"Medium"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-8D

ARC-C: 0.235. Result ID: 699a8d1a5698665ec41765b0.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"Medium"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-32

ARC-C: 0.211. Result ID: 699a8d1a5698665ec41765ac.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"Small"}. This record does not assert independent reproduction.

### Muon-8L/AdamW-8D

ARC-C: 0.205. Result ID: 699a8d1a5698665ec41765ad.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

Recorded evaluation configuration: {"Model size":"Small"}. This record does not assert independent reproduction.

### Muon-32

ARC-C: 0.204. Result ID: 699a8d1a5698665ec41765ab.

[Reporting paper](<https://www.sota2.com/research/paper/68fa7bae1e209131cea89d3b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771a1e209131cea70fcd.md>)

Recorded evaluation configuration: {"Model size":"Small"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
