Zero-shot Evaluation Suite (Food-101, CIFAR-10, CIFAR-100, SUN397, Stanford Cars, FGVC Aircraft, DTD, Oxford-IIIT Pets, Caltech-101, Flowers102, ImageNet-1K) various (test)

90.5Food-101 Top-1 Acc

Zero-shot Evaluation

Zero-shot Evaluation Suite

77.28Average Score

Zero-shot Language Understanding

Evaluation Suite Zero-shot (LMB, HellA, PIQA, ARC-E, ARC-C, WINO, Open, MMLU)

83.4ARC-E Accuracy

Language Understanding and Common Sense Reasoning

Zero-shot Evaluation Suite (PIQA, ARC-C, ARC-E, HellaS, WinoG, BoolQ, LAMBADA, C4)

79.67PIQA Accuracy

Zero-shot Question Answering and Reasoning

Evaluation Suite Zero-shot (ARC, LogiQA, Wino, CSQA, BoolQ, PIQA, MMLU)

83.88ARC

Multi-task Language Evaluation

16-task evaluation suite 1B/25B checkpoints (test)

42.22Average Score

General Language Understanding

12-task evaluation suite (test)

71.62Average Score

Aggregate Multimodal Evaluation

Evaluation Suite Average

100Average Score

Question Answering and Commonsense Reasoning

Zero-Shot Evaluation Suite (Arc-e, Arc-c, Boolq, Hellaswag, Openbookqa, Piqa, SciQ, Winogrande)

65.74ARC-E

Commonsense Reasoning

Zero-shot Evaluation Suite (HellaSwag, PIQA, Arc-E, Arc-C, WinoGrande, OBQA, SIQA, BoolQ)

38.29HellaSwag (Zero-shot)

Large Language Model Evaluation

12-task evaluation suite composite (test)

49.6Reading Comprehension Score

Language Understanding

Evaluation suite (Commonsense, Comprehension, Knowledge, Reasoning, Language tasks) (test)

43.8Average Accuracy

Zero-shot Classification

Evaluation Suite Zero-shot (PiQA, LAMBDA, ARC-e, ARC-c, HellaS)

5.23Decode Latency

Natural Language Understanding

Evaluation Suite Zero-shot (Arc-c, Arc-e, BoolQ, COPA, MMLU, OBQA, PIQA, RTE, Winogrande)

56.14ARC-c

Zero-shot Classification

Zero-shot Evaluation Suite (AC, AE, WI, QA) v1

46.2AC Score

Question Answering

Evaluation Suite Zero-shot

54.3Accuracy (Zero-shot)

Zero-shot Evaluation

Zero-shot Evaluation Suite Aggregate

73Average Accuracy (Zero-shot)