OlmoBaseEval
Benchmarks
TaskDatasetSOTA resultTrendResults
OlmoBaseEval MC Non-STEM (MMLU Humanities/Social Sci, CSQA, PiQA, SocialIQA, CoQA, DROP, Jeopardy, NaturalQs, SQuAD)
89.3Aggregate Score
34
OlmoBaseEval Code BigCodeBench, HumanEval, DeepSeek LeetCode, DS 1000, MBPP, MultiPL
54.9OlmoBaseEval Code Score
34
OlmoBaseEval Math (GSM8k, GSM Symbolic, MATH)
68.5Math Aggregate Score
34
OlmoBaseEval LBPP BBH MMLU Pro MC Deepmind Math (HeldOut)
42.6LBPP Score
24
OlmoBaseEval MCSTEM
83.4MCSTEM Score
22
OlmoBaseEval HeldOut (LBPP, BBH, MMLU Pro, etc.)
33.7LBPP Score
10