Code Generation on OlmoBaseEval Code, BigCodeBench, HumanEval, DeepSeek LeetCode, DS 1000, MBPP, MultiPL

54.9OlmoBaseEval Code ScoreSeed 36B

Evaluation Results

Method
Seed 36B2025.12	54.9	50.7	71.3	13	44	72	69.2	63.8
Qwen 2.5 32B2025.12	48.3	48.1	65.6	8	43.3	69.8	49.7	53.6
Nemo. 3 Nano2026.04	47.3	45.7	77.1	10.1	33.3	66.4	48.3	50
Qwen 32026.04	46.2	43.1	71.2	8.8	33.3	65.8	52.5	48.5
Qwen2 7B2025.12	46.1	42.5	71.7	8.3	33.1	66.2	52.3	48.4
Falcon H12026.04	45.3	40.5	61	3.5	29	63.5	59.7	59.8
Nemo-Nano 9B2025.12	43.1	43.2	71.7	6.8	30.3	62.3	40	47.5
Mistral 3.1 24B2025.12	42.4	46.4	65.5	0.1	36.3	61.9	39	47.7
Gemma 3 27B2025.12	41.6	44	62.1	5.8	34.3	60	37.7	47.2
Qwen 2.5 7B2025.12	41	39.7	66.1	5.1	35.2	55.4	40.3	45.4
Gemma 2 27B2025.12	41	43.4	57.5	4.7	29.7	61.7	40.3	49.7
Olmo 3 32B2025.12	39.7	43.7	65.8	2	29.4	59.6	36	41.5
Nemotron-H2026.04	37.1	40.3	60.1	4.1	24.7	58	32.3	40.2
Llama 3.1 70B2025.12	36.3	43.4	57.4	0.2	29.5	55.5	32.2	35.9
MiMo 7B2025.12	35.7	38.3	57	1.2	28.1	48.3	34.5	42.5
LLM 360 K2V2 70B2025.12	35.2	39.8	51.2	2.3	25.4	53.5	31.3	42.8
Olmo Hybrid2026.04	32.4	35.1	49	2.2	21.1	50.3	29.4	39.5
Olmo 32026.04	30.9	34.8	49	1.5	20.6	43.6	28.8	38.3
Marin 32B2025.12	30.8	34.5	52.3	1.3	26.3	52.1	18.5	30.5
Olmo 3 7B2025.12	30.7	34.1	49.1	1.4	20.2	43.6	28.7	38.2
Kimi Linear2026.04	30.3	44.5	72.4	1.1	34.3	58.3	1.1	0.6
Gemma 2 9B2025.12	30.2	30.9	40	1.9	23.4	49.1	27.9	38.2
Recurr. Gemma2026.04	23.7	20.2	34.5	0.6	19.1	33.7	21.6	36.4
Apertus 70B2025.12	23.3	24	32.5	1.2	17.8	37.6	18.4	31.3
Marin 8B2025.12	21.4	21.5	31.6	0.5	16.5	36.5	15.6	27.6
Llama 3.1 8B2025.12	21.2	30.7	40.4	0.1	22.2	12.1	14.5	28.3
OLMo 2 32B2025.12	20.5	22.2	29.4	0.8	20.4	37.1	10.5	23.2
Gaperon 24B2025.12	19.4	17	31.2	0	11	36.7	13	26.5
Apertus 8B2025.12	19	20.9	21.6	0.6	11.8	33.5	15.5	29.2
Granite 3.3 8B2025.12	18	0.4	0	0	22.6	48.5	22.3	32.3
Gaperon 8B2025.12	16.1	13	24.5	0	9.1	29.3	12.1	24.6
Falcon Mamba2026.04	14.6	0.2	0.1	0.1	16.3	37.5	15.7	32.1
OLMo 2 7B2025.12	10.4	8.8	16.3	0.2	10.1	21.2	4.2	12.2
xLSTM2026.04	3.1	4	13.3	0	2.3	1.3	0.2	0.3