# Zero-shot Evaluation on AI2 OLMES zero-shot Source: [Zero-shot Evaluation on AI2 OLMES zero-shot]() Collection index: [llms.txt]() Data updated: 2026-05-27T06:54:15.437Z Zero-shot performance evaluation across 10 specific tasks (including ARC-E, BoolQ, HellaSwag, etc.) using the AI2 OLMES zero-shot configuration. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Zero-shot Evaluation | | Dataset (including any stated split) | AI2 OLMES zero-shot | | Metric | ScienceQA | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [ScienceQA](): Higher is better; unit: PERCENT. - [ARC-E](): Higher is better; unit: PERCENT. - [ARC-C](): Higher is better; unit: PERCENT. - [BoolQ](): Higher is better; unit: PERCENT. - [CSQA](): Higher is better; unit: PERCENT. - [HellaSwag](): Higher is better; unit: PERCENT. - [OBQA](): Higher is better; unit: PERCENT. - [PIQA](): Higher is better; unit: PERCENT. - [SIQA](): Higher is better; unit: PERCENT. - [WinoG](): Higher is better; unit: PERCENT. - [Core 9 Accuracy](): Higher is better; unit: PERCENT. - [Commonsense Reasoning](): Higher is better; unit: PERCENT. - [Logic & Linguistics](): Higher is better; unit: PERCENT. - [Average Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | ScienceQA | Result date | Report paper | | --- | --- | --- | --- | | GEM | 35.96 | Not recorded | 6a167a67dbd7150506f7fe59 | | WebOrganizer Format | 35.06 | Not recorded | 6a167a67dbd7150506f7fe59 | | WebOrganizer Topic | 35.05 | Not recorded | 6a167a67dbd7150506f7fe59 | | GEM | 34.79 | Not recorded | 6a167a67dbd7150506f7fe59 | | WebOrganizer Topic | 34.68 | Not recorded | 6a167a67dbd7150506f7fe59 | | WebOrganizer Format | 34.44 | Not recorded | 6a167a67dbd7150506f7fe59 | | WebOrganizer Format | 34.12 | Not recorded | 6a167a67dbd7150506f7fe59 | | GEM | 34.07 | Not recorded | 6a167a67dbd7150506f7fe59 | | WebOrganizer Topic | 33.9 | Not recorded | 6a167a67dbd7150506f7fe59 | | TOS | 32.54 | Not recorded | 6a167a67dbd7150506f7fe59 | | TOS | 32.49 | Not recorded | 6a167a67dbd7150506f7fe59 | | K-Means | 32.48 | Not recorded | 6a167a67dbd7150506f7fe59 | | K-Means | 32.18 | Not recorded | 6a167a67dbd7150506f7fe59 | | TOS | 32.18 | Not recorded | 6a167a67dbd7150506f7fe59 | | K-Means | 31.63 | Not recorded | 6a167a67dbd7150506f7fe59 | ### GEM ScienceQA: 35.96. Result ID: 6a1694fefbc373592c0f2b63. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"Perf","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### WebOrganizer Format ScienceQA: 35.06. Result ID: 6a1694fefbc373592c0f2b62. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"Perf","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### WebOrganizer Topic ScienceQA: 35.05. Result ID: 6a1694fefbc373592c0f2b61. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"Perf","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### GEM ScienceQA: 34.79. Result ID: 6a1694fefbc373592c0f2b5e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"DoReMi","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### WebOrganizer Topic ScienceQA: 34.68. Result ID: 6a1694fefbc373592c0f2b5c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"DoReMi","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### WebOrganizer Format ScienceQA: 34.44. Result ID: 6a1694fefbc373592c0f2b5d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"DoReMi","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### WebOrganizer Format ScienceQA: 34.12. Result ID: 6a1694fefbc373592c0f2b67. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"RegMix","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### GEM ScienceQA: 34.07. Result ID: 6a1694fefbc373592c0f2b68. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"RegMix","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### WebOrganizer Topic ScienceQA: 33.9. Result ID: 6a1694fefbc373592c0f2b66. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"RegMix","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### TOS ScienceQA: 32.54. Result ID: 6a1694fefbc373592c0f2b60. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"Perf","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### TOS ScienceQA: 32.49. Result ID: 6a1694fefbc373592c0f2b5b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"DoReMi","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### K-Means ScienceQA: 32.48. Result ID: 6a1694fefbc373592c0f2b5f. [Reporting paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"Perf","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### K-Means ScienceQA: 32.18. Result ID: 6a1694fefbc373592c0f2b5a. [Reporting paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"DoReMi","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### TOS ScienceQA: 32.18. Result ID: 6a1694fefbc373592c0f2b65. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"RegMix","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. ### K-Means ScienceQA: 31.63. Result ID: 6a1694fefbc373592c0f2b64. [Reporting paper]() Recorded evaluation configuration: {"Data Mixing Strategy":"RegMix","Evaluation Protocol":"zero-shot","Model Architecture":"LLaMA-style Transformer","Parameters":"1.1B","Training Tokens":"25B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.