Factual code question answering on CodeSimpleQA English 1.0

60ADSchatgpt-4o-latest

Evaluation Results

Method
chatgpt-4o-latest2025.12	60	50	0	63.6	58.8	30	62.5	40.6	48.3	75	66.7	59.3	23.1	25	63.9	45	43.1	58.1	36.4	0	51.4	50.5
gpt-5-mini2025.12	50	60	50	70.6	53.8	57.1	47.1	50	40.5	50	54.5	75	41.7	25	68.2	50	53.7	58.4	46.7	100	56.5	53.5
o3-mini-2025-01-312025.12	50	66.7	50	66.7	52.9	40	54.2	50	48.3	58.3	57.7	50	30.8	62.5	58.3	40	35.3	51.4	37.2	100	45.2	47.5
gpt-oss-20b2025.12	50	16.7	28.6	27.3	17.6	20	38.3	28.1	36.2	43.5	57.7	28.6	23.1	37.5	47.2	30	39.6	36.7	27.8	50	38	35.8
Qwen3-235B-A22B2025.12	50	50	25	63.6	29.4	40	29.2	50.8	36.2	58.3	50	42.9	23.1	25	45.1	55	39.2	49.1	31	100	40.8	42
Qwen3-235B-A22B-Thinking-25072025.12	50	50	0	54.5	58.8	40	50	34.4	43.1	41.7	61.5	50	30.8	25	52.8	30	25.5	50.2	32.1	50	41.7	42.5
Qwen3-Coder-480B-A35B-Instruct2025.12	50	33.3	0	28.6	52.9	30	46.8	19.4	40	58.3	39.1	64.3	48	40	70.4	45	47.1	44.8	28	100	43.7	42.6
claude-sonnet-4-20250514-thinking2025.12	44.4	25	50	50	36.4	61.5	58.3	54.5	40.4	35.3	44.4	50	36.4	46.2	45.3	62.5	49.3	55.8	43	100	53.2	49.6
GLM-4.5-Air2025.12	44.4	36.4	0	57.1	24.2	40	31.1	39.3	33	26.1	34	51.9	24	13.3	58	30.8	38.3	37.5	26.2	100	39.5	36.5
Qwen3-Coder-30B-A3B-Instruct2025.12	42.1	50	0	38.1	35.3	20	38.3	34.9	29.6	8.7	29.2	57.1	16	0	39.4	30	33.3	33.8	23.9	100	30.4	31.3
Qwen3-235B-A22B-Instruct-25072025.12	42.1	50	0	54.5	47.1	40	41.7	37.5	44.8	58.3	42.3	35.7	30.8	37.5	52.8	40	35.3	43.3	34	100	40.6	41.2
o32025.12	40	33.3	50	72.7	82.4	60	70.8	59.4	69	66.7	57.7	78.6	30.8	62.5	72.2	65	60.8	62.5	54.3	100	58	61.2
o4-mini2025.12	40	50	25	60	58.8	30	50	56.7	57.9	50	68	42.9	38.5	50	65.7	40	51.1	55.8	39.3	100	47.9	51.1
Qwen2.5-Coder-7B-Instruct2025.12	40	16.7	0	9.1	17.6	20	42.6	21.9	24.1	16.7	34.6	29.6	8	12.5	44.4	25	26	29.5	15	100	23.2	25.2
Seed-Coder-8B-Instruct2025.12	40	18.2	0	18.2	11.8	10	37.5	12.5	25.9	25	26.9	29.6	0	0	33.3	30	23.8	26.5	10.8	0	29.6	23.8
Qwen3-8B2025.12	40	36.4	0	18.2	23.5	10	33.3	29	22.6	34.8	28.6	28.6	0	0	37.1	45	29.7	32.7	9.8	50	30.3	27.1
Qwen2.5-Coder-14B-Instruct2025.12	40	16.7	0	36.4	29.4	10	45.8	21.9	24.1	50	19.2	51.9	15.4	12.5	50	30	30	32.8	19.3	50	32.1	30
Qwen2.5-14B-Instruct2025.12	40	0	0	27.3	23.5	30	33.3	28.1	27.6	33.3	42.3	29.6	7.7	25	38.9	20.5	25.7	27.8	12.9	100	30.6	27.2
Qwen3-14B2025.12	40	36.4	25	36.4	23.5	10	41.7	31.7	26.3	41.7	52	35.7	7.7	0	50.7	50	23.5	31.1	21.6	50	31.1	30.9
Qwen3-30B-A3B-Instruct-25072025.12	40	33.3	0	36.4	29.4	30	45.8	25	34.5	58.3	42.3	50	38.5	12.5	52.8	35	31.4	36.9	17	100	37.1	34.7
Qwen3-30B-A3B2025.12	40	33.3	0	45.5	23.5	20	37.5	22.6	31.3	41.7	43.1	64.3	23.1	0	56.3	35	37.3	33.7	16.3	100	33.5	32.7
QwQ-32B2025.12	40	36.4	25	63.6	29.4	30	41.7	34.9	28.6	17.4	34.6	42.9	15.4	0	53.5	30	29.7	36.2	22.6	50	39.8	34.4
Qwen2.5-72B-Instruct2025.12	40	33.3	25	45.5	41.2	10	45.8	34.4	37.9	41.7	46.2	37	7.7	50	52.8	25	37.6	44.9	28.9	50	40.1	38.9
gpt-oss-120b2025.12	40	50	0	54.5	29.4	40	45.8	53.1	50	41.7	61.5	64.3	23.1	37.5	52.8	35	52.9	51.2	34	100	46.2	46.6
DeepSeek-R12025.12	40	33.3	50	54.5	35.3	30	54.2	50	41.4	41.7	61.5	42.9	38.5	12.5	47.2	55	51	53.5	31.9	100	46.2	46.2
DeepSeek-R1-05282025.12	40	50	25	54.5	35.3	40	58.3	50	36.2	50	53.8	57.1	46.2	37.5	61.1	40	37.3	50.6	32.4	100	42.1	44.5
DeepSeek-V32025.12	40	33.3	25	54.5	29.4	20	58.3	46.9	43.1	66.7	46.2	50	30.8	12.5	52.8	40	33.3	45.6	26.6	50	40	40.7
DeepSeek-V3-03242025.12	40	33.3	50	63.6	41.2	40	54.2	50	36.2	66.7	50	64.3	23.1	12.5	50	50	37.3	44.4	35.1	50	44.5	43.4
GLM-4.52025.12	40	66.7	25	66.7	29.4	40	45.8	66.7	43.1	50	57.7	50	30.8	12.5	47.2	50	35.6	47.5	41.9	50	43.1	45
Qwen2.5-Coder-32B-Instruct2025.12	40	16.7	0	54.5	47.1	10	37.5	28.1	31.3	33.3	39.2	51.9	7.7	25	52.8	25	29.4	37.5	26.6	100	33.5	33.9
CodeSimpleQA-RFT2025.12	36.4	25	0	63.6	41.2	20	33.3	34.4	34.5	16.7	53.8	51.9	15.4	25	66.7	25	33.7	41.6	34.2	50	41.7	39.2
gpt-52025.12	35.3	20	100	66.7	69.2	80	62.9	78.8	48.8	81.8	57.1	100	40	71.4	65.2	66.7	54.3	68.4	58	100	64.2	62.9
claude-sonnet-4-202505142025.12	33.3	66.7	50	35.3	44.4	36.4	61.5	57.1	50	55.6	50	63.6	16.7	61.5	68.1	62.5	51.9	53.3	37.1	0	51.6	50.3
Llama-3.1-8B-Instruct2025.12	31.6	33.3	25	38.1	23.5	10	36.4	16.1	23.2	27.3	8.9	21.4	7.7	40	50.7	5.1	32	29.6	14.6	100	21	24.4
Qwen3-4B-Thinking-25072025.12	30	33.3	25	27.3	23.5	20	25	22.2	24.6	16.7	38.5	28.6	8	0	31	20	21.6	31.1	12.9	50	25.7	24.7
Qwen3-30B-A3B-Thinking-25072025.12	30	16.7	25	36.4	23.5	20	34	28.6	27.6	41.7	43.1	50	16	0	44.4	35	31.7	32.2	16	50	35.4	30.9
Qwen2.5-32B-Instruct2025.12	30	16.7	25	47.6	41.2	30	29.2	28.6	34.5	26.1	42.3	28.6	0	25	50	15	30	39.2	17.2	100	33.6	32.3
DeepSeek-R1-Distill-Qwen-32B2025.12	30	33.3	0	36.4	23.5	20	33.3	12.5	24.1	26.1	34.6	21.4	0	0	52.8	15.4	31.4	29.5	19.1	50	30.1	27.2
Qwen3-32B2025.12	30	33.3	25	57.1	29.4	20	42.6	50	24.3	41.7	47.1	35.7	15.4	37.5	37.1	25.6	31.7	45.9	19.4	50	37	35.6
Seed-OSS2025.12	30	16.7	25	27.3	29.4	20	33.3	34.9	26.1	16.7	29.2	28.6	24	0	42.3	30	35.3	33.1	25.7	50	32.7	30.7
Qwen3-Coder-480B2025.12	30	18.2	0	11.1	43.7	27.3	42.9	13.8	35.5	60.9	34.8	64.3	34.8	42.9	64.5	32.4	33.1	39.9	32.6	100	42.4	38.2
DeepSeek-V3.12025.12	30	0	50	54.5	41.2	30	45.8	46.9	43.1	50	38.5	50	15.4	25	47.2	20	41.6	49.4	30.9	50	38.2	40.5
gpt-4o-2024-11-202025.12	28.6	50	33.3	73.7	50	33.3	62.5	42.3	39.5	66.7	68.4	54.5	20	0	68	35.3	44.4	51.5	36.9	50	45.5	46.7
CodeSimpleQA-RL2025.12	28.6	50	0	63.6	35.3	20	37.5	31.7	43.1	33.3	58.8	50	40	26.7	62	45	37.6	46.5	29.9	100	44.6	42.3
Qwen2.5-0.5B-Instruct2025.12	21.1	0	0	36.4	12.1	0	8.7	3.2	1.8	8.7	3.9	7.4	0	26.7	22.9	5.1	12	15.5	1.1	50	14.9	11
Qwen2.5-Coder-0.5B-Instruct2025.12	21.1	20	0	18.2	17.6	0	9.1	3.2	3.6	9.5	7.8	7.1	8	0	39.4	0	22.4	16.8	5.7	0	16.5	13.5
Qwen3-4B2025.12	21.1	16.7	0	38.1	17.6	20	25.5	22.2	19.1	50	28	35.7	0	12.5	42.9	10.5	35.6	30.1	13	50	28.7	26
Qwen3-1.7B2025.12	21.1	50	0	18.2	23.5	20	29.2	21.9	25.9	25	27.5	21.4	0	12.5	33.3	0	19.8	23.3	11.7	100	19.8	20.8
Qwen3-0.6B2025.12	20	16.7	0	0	17.6	0	12.8	6.3	10.7	8.3	8.2	7.1	0	0	31	15	18	11.7	3.3	50	13.8	11.7
Qwen2.5-Coder-1.5B-Instruct2025.12	20	16.7	0	28.6	12.1	0	30.4	19	8.8	16.7	15.4	21.4	0	0	44.4	10	25.7	22.1	10.8	100	24.4	20.1
Qwen2.5-1.5B-Instruct2025.12	20	16.7	0	18.2	23.5	0	20.8	9.4	12.1	25	15.4	14.8	0	25	33.3	5.3	15.8	21.4	4.3	50	25.2	17.8
DeepSeek-R1-Distill-Qwen-1.5B2025.12	20	16.7	0	0	24.2	0	12.8	9.5	3.6	8.7	3.8	0	0	0	11.3	0	2	14.6	2.1	0	5.9	7.4
Qwen2.5-Coder-3B-Instruct2025.12	20	54.5	0	27.3	30.3	10	37.5	18.8	15.7	26.1	11.8	21.4	0	25	44.4	15	23.8	25.7	15	50	16.9	21.3
Qwen2.5-3B-Instruct2025.12	20	18.2	0	0	17.6	20	29.8	15.6	13.8	16.7	19.6	14.3	0	0	33.3	10.3	20	22.2	9.7	100	21.5	18.6
DeepSeek-R1-Distill-Qwen-7B2025.12	20	0	0	18.2	18.2	0	20.8	12.5	8.7	0	19.2	0	0	0	13.9	0	14	17.6	4.3	50	13.3	12.1
Qwen2.5-7B-Instruct2025.12	20	16.7	0	19	35.3	20	34	28.1	27.6	16.7	26.9	29.6	7.7	0	36.1	10	23.8	22.7	16	100	22.7	23.1
Hunyuan-7B-Instruct2025.12	20	0	0	9.5	17.6	20	34	9.4	10.6	8.7	12.2	7.1	15.4	0	29	15.4	11.9	20.5	7.5	0	18.7	15.9
DeepSeek-R1-Distill-Llama-8B2025.12	20	0	0	9.1	11.8	10	25	18.8	15.5	25	19.2	14.3	0	0	27.8	10	17.6	19.4	6.4	50	15.1	15.6
DeepSeek-R1-Distill-Qwen-14B2025.12	20	16.7	0	9.1	17.6	0	37.5	25	24.1	16.7	31.4	7.1	8	12.5	38.9	20	23.5	25.8	13.8	50	24.9	23
DeepSeek-R1-Distill-Llama-70B2025.12	20	33.3	25	36.4	17.6	40	41.7	31.2	37.9	25	35.3	35.7	7.7	12.5	50	30	35.3	34.6	23.5	50	29.6	31.9
DeepSeek-R1-0528-Qwen3-8B2025.12	11.1	18.2	0	36.4	24.2	10	39.1	13.8	18.3	26.1	31.4	30.8	0	0	39.4	27	25	30.3	12.6	66.7	27.9	24.8
Llama-3.2-1B-Instruct2025.12	0	33.3	0	0	23.5	0	18.2	9.7	7	19	4.1	0	8	25	22.9	0	13.9	10.7	4.5	0	14.1	10.9
Llama-3.2-3B-Instruct2025.12	0	16.7	0	20	23.5	0	22.2	3.4	14.3	40	8.9	14.3	0	40	34.3	0	16	19	8.2	0	16.5	15.6