# Zero-shot LLM Evaluation Suite (AE, AC, SciQ, MM, MM-P, HS, OBQA, PIQA, RACE, WG, CSQA, AGI)

Source: [Zero-shot LLM Evaluation Suite (AE, AC, SciQ, MM, MM-P, HS, OBQA, PIQA, RACE, WG, CSQA, AGI)](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-19T11:38:18.375Z

Evaluation of zero-shot performance across a comprehensive suite of 12 language understanding tasks: AE, AC, SciQ, MM, MM-P, HS, OBQA, PIQA, RACE, WG, CSQA, and AGI.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Zero-shot Language Understanding |
| Dataset (including any stated split) | LLM Evaluation Suite (AE, AC, SciQ, MM, MM-P, HS, OBQA, PIQA, RACE, WG, CSQA, AGI) zero-shot |
| Metric | AE Score |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 2 of 2 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [AE Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=AE%20Score>): Higher is better; unit: PERCENT.

- [AC Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=AC%20Score>): Higher is better; unit: PERCENT.

- [SciQ Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=SciQ%20Score>): Higher is better; unit: PERCENT.

- [MM Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=MM%20Score>): Higher is better; unit: PERCENT.

- [MM-P Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=MM-P%20Score>): Higher is better; unit: PERCENT.

- [HS Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=HS%20Score>): Higher is better; unit: PERCENT.

- [OBQA Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=OBQA%20Score>): Higher is better; unit: PERCENT.

- [PIQA Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=PIQA%20Score>): Higher is better; unit: PERCENT.

- [RACE Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=RACE%20Score>): Higher is better; unit: PERCENT.

- [WG Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=WG%20Score>): Higher is better; unit: PERCENT.

- [CSQA Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=CSQA%20Score>): Higher is better; unit: PERCENT.

- [AGI Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=AGI%20Score>): Higher is better; unit: PERCENT.

- [Average Score](<https://www.sota2.com/research/sota/zero-shot-llm-evaluation-suite-ae-ac-sciq-mm-mm-p-hs-obqa-piqa-race-wg-csqa-agi.md?metric=Average%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | AE Score | Result date | Report paper |
| --- | --- | --- | --- |
| AMO | 67.97 | Not recorded | 6a0beef0dbc836b87d82c8e0 |
| Muon-KJ | 65.32 | Not recorded | 6a0beef0dbc836b87d82c8e0 |

### AMO

AE Score: 67.97. Result ID: 6a0c4ba6fbc373592c036d48.

[Reporting paper](<https://www.sota2.com/research/paper/6a0beef0dbc836b87d82c8e0.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a0beef0dbc836b87d82c8e0.md>)

Recorded evaluation configuration: {"Backbone":"Llama3.1-1.4B","Pre-training token budget":"2x Chinchilla optimal ratio"}. This record does not assert independent reproduction.

### Muon-KJ

AE Score: 65.32. Result ID: 6a0c4ba6fbc373592c036d47.

[Reporting paper](<https://www.sota2.com/research/paper/6a0beef0dbc836b87d82c8e0.md>)

Recorded evaluation configuration: {"Backbone":"Llama3.1-1.4B","Pre-training token budget":"2x Chinchilla optimal ratio"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.