Generative Question Answering on Bolmo Evaluation Suite GenQA 7B

81.6GenQA AverageLlama 3.1 70B

Evaluation Results

Method
Llama 3.1 70B2025.12	81.6	88.4	91.7	79.6	92.4	78.3	84	53.1	92.9	73.9
Marin 32B2025.12	80.3	87.2	90.5	76.7	91.1	76.5	80.5	55.1	94.4	70.7
Olmo 3 32B2025.12	79.8	84.8	90.3	75.7	93.5	80.9	75.3	49	94.5	74.1
OLMo 2 32B2025.12	79.1	87.5	89.4	77	88.7	76.3	79.1	51.4	94	68.7
Nemo. 3 Nano2026.04	78.1	86	88.7	75.9	92.5	75	77.2	45.7	94	67.7
Mistral 3.1 24B2025.12	78	86.2	90.8	79.3	91.9	74.9	80.3	45.1	92.6	61.1
LLM 360 K2V2 70B2025.12	77.1	87.6	88.9	76.8	90.6	69.7	79.8	47.6	91.2	61.5
Seed 36B2025.12	76	84.8	89.3	76.1	96	76.1	77.4	30.7	89.1	64.4
Marin 8B2025.12	75.9	84	88.6	73.9	85.6	73	72.7	42.6	93.4	69.5
Kimi Linear2026.04	75.7	85	88.2	77.2	93.1	72.2	77.1	25.9	91.8	70.3
Gemma 2 9B2025.12	75.6	81.8	88.8	76.3	89.3	68.2	75.1	40.4	88.8	71.5
Apertus 70B2025.12	75	84.5	87.7	74.8	87.5	56.3	77.2	43.1	90.7	72.8
Gemma 3 27B2025.12	73.5	86	91.3	77.5	94.9	75.9	82.1	49.2	92.4	12.4
Llama 3.1 8B2025.12	73.1	81.5	87.3	75.5	88	59.5	70.9	36.7	89.2	69
Gemma 2 27B2025.12	72.9	86.7	90.8	76.9	93.2	73.2	80.7	47.1	93	14.9
Olmo Hybrid2026.04	72.9	79	86.2	70.2	89.7	72.9	64	34.8	92.1	67.4
Olmo 3 7B2025.12	72.5	77.7	85.7	68.9	89.5	71.5	60.4	32.6	93.5	72.8
Olmo 32026.04	72.5	77.7	85.6	68.8	89.5	71.5	60.3	32.5	93.6	72.8
OLMo 2 7B2025.12	72.4	82.2	87.4	70.5	82.2	61.5	70.8	37.4	91.5	68.3
Nemo-Nano 9B2025.12	71.8	80.2	86.2	67.9	91.4	71.4	64.9	31.2	92.3	60.4
Falcon H12026.04	71.7	80.9	87.9	73	93.3	69.4	61.1	27.4	91.4	60.7
MiMo 7B2025.12	71.4	80.6	86.5	73.1	89.7	69.3	65.6	33.1	90.3	54.4
Nemotron-H2026.04	71.2	84.3	89.6	76	90.2	65.7	69.7	37.7	87.9	39.9
Qwen2 7B2025.12	71.1	80.5	86.4	73	93.5	57.2	65.1	33.8	89.2	61.6
Qwen 32026.04	71.1	80.6	86.4	72.8	93.4	57.2	65.1	33.8	89	61.8
Apertus 8B2025.12	69	81	85.8	70.9	83.8	37.1	70.1	35	89.6	67.4
Qwen 2.5 32B2025.12	68.5	86.3	87.5	76.2	94.2	53.7	74	39.3	64.9	40.4
Recurr. Gemma2026.04	68.5	81.3	86.3	72.2	83	44	66.4	30.6	85.9	66.6
Falcon Mamba2026.04	68.5	81.9	88.3	80.9	86.1	41.6	67.1	35.1	75.9	59.7
Granite 3.3 8B2025.12	67.8	83.7	89.4	76	88.7	38.4	69.7	37	89.6	37.8
Qwen 2.5 7B2025.12	67.5	81	86	70.3	91.4	56.7	63	31.2	87	40.5
Gaperon 24B2025.12	65.3	75.2	80.3	58.3	83.2	59.4	58.9	33.5	89.3	49.8
Gaperon 8B2025.12	63.3	73.9	76.4	67	80.5	54.9	55.5	28.8	86	46.7
xLSTM2026.04	34.8	61.1	62.1	19.7	60.9	24.4	9.3	1.8	33.9	40.4
Olmo 3 7B2025.12	0.724	0.778	0.857	0.68	0.9	0.715	0.603	0.326	0.935	0.727
TFree-Hat 7B2025.12	0.713	0.828	0.882	0.705	0.896	0.486	0.683	0.343	0.886	0.711
Bolmo 7B2025.12	0.709	0.788	0.855	0.711	0.896	0.652	0.568	0.286	0.916	0.705
BLT 7B2025.12	0.684	0.811	0.882	0.728	0.845	0.388	0.673	0.292	0.852	0.686
EvaByte 6.5B2025.12	0.414	0.701	0.782	0.629	0.827	0.078	0.131	0.054	0.359	0.167