Code Question Answering on CodeSimpleQA Chinese

100ADSQwen3-30B-A3B-Thinking-2507

Evaluation Results

Method
Qwen3-30B-A3B-Thinking-25072025.12	100	68.4	42.9	80	46.7	47.1	51.4	63.2	51.4	57.1	31.2	64.7	46.2	53.8	45.4	39.3	40.8	47.5	42.2	58.8	40.2	46.7
Qwen2.5-1.5B-Instruct2025.12	85.7	26.3	14.3	40	23.7	12.1	21.6	47.4	14.5	28.6	6.3	24.2	15.4	53.8	20.4	21.8	21.1	16.9	13.8	22.2	16.3	20
gpt-4o-2024-11-202025.12	75	78.6	60	87.5	50	41.7	51	73.3	44	75	32	38.5	60	55.6	46.6	50	55.1	53.7	48	40	43.2	50.7
o32025.12	75	68.4	71.4	80	70	58.8	67.6	68.4	61.8	57.1	56.2	58.8	46.2	61.5	63.3	57.1	73.2	59.1	57.3	66.7	57	62.1
claude-sonnet-4-20250514-thinking2025.12	75	73.3	57.1	100	53.8	47.6	58.1	80	54.2	44.4	51.2	31.6	36.4	70	59.8	60	64.5	46.2	50.9	76.9	47.1	55.4
gpt-52025.12	75	90.9	80	100	77.8	62.5	75.5	84.6	60	50	57.1	50	66.7	55.6	74.2	62.1	88.4	58.1	67.4	80	47.5	67.2
DeepSeek-R1-Distill-Qwen-1.5B2025.12	75	16.7	0	31.6	6.9	6.2	5.8	17.6	3	14.3	3.6	11.8	7.7	30.8	2.3	10.9	0	3.8	2.6	11.8	3.2	6.2
Qwen3-4B-Thinking-25072025.12	75	78.9	14.3	70	43.3	35.3	41.7	47.4	45.7	42.9	25	42.4	32	38.5	30.6	35.7	39.4	31.6	29.6	66.7	39.2	38.3
Qwen2.5-7B-Instruct2025.12	75	48.6	28.6	42.1	23.3	17.6	27	36.8	22.9	42.9	15.6	30.3	15.4	46.2	27.1	36.4	28.4	24	20.9	22.2	22.3	26.3
Seed-Coder-8B-Instruct2025.12	75	21.1	14.3	30	40	24.2	21.6	32.4	22.9	42.9	9.4	35.3	16	46.2	35.4	35.7	32.9	32.6	26.3	33.3	31.5	29.8
DeepSeek-R1-0528-Qwen3-8B2025.12	75	32.4	30.8	21.1	38.5	24.2	36.1	48.5	26.9	50	32.3	44.4	33.3	24	29.2	34.6	32.1	27.2	21.3	35.3	24.6	30
Qwen2.5-14B-Instruct2025.12	75	47.4	28.6	50	37.9	29.4	37.8	52.6	43.5	42.9	12.9	41.2	15.4	30.8	31.2	32.7	38	33.9	23.2	55.6	23.1	32.5
gpt-oss-20b2025.12	75	47.4	42.9	70	54.2	35.3	43.2	47.4	43.5	42.9	25.4	42.4	46.2	38.5	43.3	39.3	49.6	40.7	30.9	55.6	50	43.3
Qwen2.5-32B-Instruct2025.12	75	52.6	42.9	60	36.7	29.4	37.8	47.4	17.1	14.3	25	35.3	32	46.2	39.2	39.3	45.1	35	29.4	35.3	39.4	36.7
DeepSeek-R1-Distill-Qwen-32B2025.12	75	42.1	14.3	30	30	17.6	29.7	59.5	31.9	28.6	21.9	25	23.1	48	41.2	28.6	34	30.1	29.6	47.1	35.4	32.8
QwQ-32B2025.12	75	63.2	30.8	40	40	41.2	41.7	75.7	50	71.4	36.1	41.2	64	61.5	45.8	32.7	51.1	35.8	33.8	47.1	43.7	44
Seed-OSS2025.12	75	52.6	0	50	36.7	36.4	43.8	52.6	48.6	42.9	16.1	36.4	32	46.2	37.5	32.1	39.7	32.8	27.5	47.1	38	36.8
DeepSeek-R1-Distill-Llama-70B2025.12	75	48.6	15.4	30	36.7	29.4	32.4	47.4	49.3	28.6	25.4	41.2	46.2	38.5	41.7	28.6	46.5	34.7	28	35.3	44	38
Qwen3-235B-A22B-Instruct-25072025.12	75	68.4	42.9	70	53.3	35.3	56.8	78.9	51.4	57.1	28.1	50	46.2	53.8	49	60.7	56.3	48.3	43.9	55.6	42	49.9
Qwen3-235B-A22B-Thinking-25072025.12	75	78.9	28.6	90	53.3	47.1	62.2	68.4	43.5	71.4	37.5	58.8	53.8	46.2	53.1	50	57.7	50.8	40.5	44.4	56.3	52.9
Qwen3-Coder-480B-A35B-Instruct2025.12	75	68.4	42.9	70	50	36.4	48.6	64.9	37.7	57.1	34.4	35.3	61.5	61.5	49.5	47.3	60.6	45.2	42	55.6	40	47.7
DeepSeek-V32025.12	75	57.9	42.9	80	46.7	52.9	51.4	73.7	48.6	71.4	40.6	54.5	46.2	61.5	44.9	46.4	57.7	46.1	40.2	44.4	48.2	49.6
DeepSeek-V3-03242025.12	75	84.2	57.1	90	60	47.1	56.8	63.2	54.3	71.4	34.4	41.2	38.5	53.8	53.6	42.9	59.2	46.1	45.1	55.6	47	51.6
DeepSeek-V3.12025.12	75	78.9	57.1	80	56.7	29.4	54.1	68.4	54.3	42.9	21.9	47.1	38.5	61.5	44.9	57.1	60.6	47.7	39	66.7	46.2	49.8
claude-sonnet-4-202505142025.12	66.7	61.5	28.6	85.7	60	51.9	51.9	53.3	54.5	36.4	38.1	33.3	33.3	42.9	58.3	70.3	64.3	54.7	44.1	76.9	49.4	53.5
Qwen3-8B2025.12	57.1	57.9	46.2	52.6	37.3	29.4	43.2	57.9	45.7	85.7	28.6	29.4	15.4	38.5	31.2	35.7	44	33	30.9	47.1	29.7	36.8
chatgpt-4o-latest2025.12	50	73.7	46.2	70	63.3	54.5	51.4	78.9	51.4	42.9	43.8	52.9	46.2	53.8	49	50	59.2	49.4	45.1	55.6	42	51.4
o3-mini-2025-01-312025.12	50	68.4	42.9	100	66.7	47.1	45.9	73.7	51.4	85.7	34.9	47.1	53.8	53.8	61.9	50	57.7	47.2	47.9	47.1	52.3	53.4
o4-mini2025.12	50	72.2	50	100	63.3	46.7	54.3	84.2	58.8	83.3	41.4	60	50	53.8	56.5	44.4	70.6	46.1	40	77.8	54.6	55.5
Qwen2.5-Coder-1.5B-Instruct2025.12	50	16.2	28.6	21.1	16.9	5.9	10.8	27	17.1	14.3	9.4	18.2	23.1	30.8	16.5	25	25.5	19.2	12.7	11.1	16.5	17.9
Qwen2.5-Coder-3B-Instruct2025.12	50	31.6	42.9	11.1	27.1	11.8	30.1	36.8	23.5	28.6	9.7	36.4	23.1	61.5	28.9	25	25.9	22.6	19	35.3	25.6	25.5
Qwen2.5-3B-Instruct2025.12	50	16.2	14.3	40	13.6	12.1	24.7	36.8	14.3	14.3	9.4	42.4	24	46.2	14.4	32.7	17	20.7	16.2	23.5	18.1	20
Qwen3-4B2025.12	50	54.1	28.6	63.2	40	24.2	35.6	42.1	31.9	57.1	25	23.5	30.8	53.8	33	28.6	36.5	31	31.2	50	33.5	34.5
DeepSeek-R1-Distill-Qwen-7B2025.12	50	31.6	14.3	30	13.3	11.8	16.4	31.6	8.6	0	9.5	12.1	8	32	6.2	7.3	10.1	12.6	5	44.4	5	11.6
Qwen2.5-Coder-7B-Instruct2025.12	50	31.6	28.6	40	30.5	17.6	16.4	36.8	20.3	42.9	15.6	41.2	15.4	23.1	26.8	25	34.5	28.2	25.9	11.1	31.5	27.6
Qwen3-1.7B2025.12	50	52.6	14.3	50	36.7	17.6	19.2	57.9	23.2	42.9	21.9	17.6	24	48	26.5	18.2	21.4	13.5	18.3	47.1	16	23.4
Hunyuan-7B-Instruct2025.12	50	31.6	28.6	40	26.7	12.1	29.7	21.6	14.3	14.3	19	11.8	23.1	38.5	14.4	25	23	25	13.7	11.1	18.3	21
DeepSeek-R1-Distill-Llama-8B2025.12	50	21.1	14.3	31.6	16.7	17.6	8.7	31.6	11.9	28.6	3.2	30.3	33.3	38.5	14.9	10.9	18.4	16.2	12.8	22.2	15.4	16.7
Qwen2.5-Coder-14B-Instruct2025.12	50	47.4	42.9	31.6	30	42.4	29.7	36.8	28.6	57.1	19	41.2	23.1	46.2	38.8	46.4	42.3	31.5	32.1	33.3	30.2	34.5
DeepSeek-R1-Distill-Qwen-14B2025.12	50	48.6	28.6	50	30	32.3	30.1	42.1	31.9	42.9	9.4	24.2	15.4	61.5	39.6	25.5	35.7	32.6	25.8	35.3	29.1	31.6
Qwen3-30B-A3B-Instruct-25072025.12	50	68.4	42.9	60	36.7	35.3	51.4	63.2	34.3	57.1	21.9	41.2	38.5	46.2	42.9	53.6	45.1	39.5	37.8	44.4	40	42.3
Qwen3-30B-A3B2025.12	50	42.1	57.1	60	43.3	35.3	49.3	57.9	42.9	57.1	25	29.4	46.2	53.8	46.3	50	40.8	33.3	32.5	35.3	43.4	41
Qwen3-Coder-30B-A3B-Instruct2025.12	50	68.4	57.1	50	33.3	37.5	45.9	63.2	37.1	57.1	28.1	23.5	53.8	53.8	24.5	42.9	45.1	30.3	33.1	55.6	36	38.5
Qwen3-32B2025.12	50	57.9	28.6	70	36.7	29.4	49.3	63.2	51.4	71.4	28.1	47.1	38.5	53.8	46.9	47.3	56.7	37.5	39.8	44.4	36.5	44.1
Qwen2.5-72B-Instruct2025.12	50	63.2	30.8	50	30	29.4	43.2	63.2	42.9	42.9	21.9	42.4	23.1	53.8	41.2	50	45.7	40.7	30.7	44.4	41.4	40.5
gpt-oss-120b2025.12	50	73.7	42.9	70	63.3	41.2	54.1	63.2	62.9	85.7	46.9	47.1	38.5	53.8	57.1	50	62	50.6	46.3	55.6	51	54.1
Qwen3-235B-A22B2025.12	50	52.6	42.9	80	43.3	35.3	51.4	68.4	48.6	71.4	31.2	36.4	38.5	38.5	49.5	39.3	57.7	42	31.9	55.6	39	44.5
DeepSeek-R12025.12	50	63.2	28.6	60	43.3	35.3	48.6	57.9	57.1	71.4	37.5	41.2	46.2	30.8	52.1	50	52.1	50.8	41.5	66.7	46	48.2
DeepSeek-R1-05282025.12	50	47.4	28.6	70	50	35.3	54.1	68.4	51.4	57.1	25	52.9	46.2	69.2	53.1	39.3	49.3	47.2	39.3	44.4	48.2	47.5
GLM-4.52025.12	50	63.2	42.9	90	53.3	41.2	51.4	63.2	54.3	71.4	25	41.2	53.8	46.2	51	46.4	56.7	52	47.9	58.8	49.2	50.9
GLM-4.5-Air2025.12	50	63.2	42.9	90	46.7	5.9	48.6	52.6	45.7	57.1	28.1	47.1	38.5	38.5	49	29.1	56.3	46.1	34.1	58.8	42.4	44.3
Qwen2.5-Coder-32B-Instruct2025.12	50	63.2	28.6	52.6	40	36.4	40.5	68.4	34.8	42.9	28.1	42.4	15.4	53.8	43.3	42.9	44	38.2	35.4	55.6	35	40
CodeSimpleQA-RFT2025.12	50	70.6	50	41.7	46.7	41.2	38.5	52.4	42.9	28.6	27.3	41.2	20	53.8	31.4	64.3	48.6	39.3	38.4	55.6	42.2	42.3
CodeSimpleQA-RL2025.12	50	78.8	37.5	60.9	43.3	38.7	39	57.1	48.6	28.6	37.5	52.9	33.3	61.5	35.6	43.6	53.1	45.9	39	55.6	43.5	45.2
gpt-5-mini2025.12	40	78.6	100	85.7	58.5	50	58.3	69.6	57.8	100	44.4	71	55.6	66.7	68.6	46.7	66.7	45.8	52.7	0	54	57.8
Qwen3-14B2025.12	28.6	68.4	42.9	70	36.7	35.3	45.9	47.4	31.4	85.7	31.2	36.4	61.5	30.8	42.9	44.4	46.8	35.6	36	55.6	33.2	40.5
Qwen2.5-0.5B-Instruct2025.12	25	11.8	15.4	0	13.8	0	24.7	21.6	5.7	30.8	12.9	24.2	0	24	12.5	25.5	12.8	12.9	10.1	22.2	11.3	13.5
Qwen2.5-Coder-0.5B-Instruct2025.12	25	5.4	0	11.1	14.5	6.5	8.5	21.1	2.9	0	9.8	13.3	16.7	30.8	14.9	7.3	14.3	14	16.6	0	15.1	13
Llama-3.2-3B-Instruct2025.12	25	21.1	14.3	10	16.9	11.8	10.8	15.8	14.3	14.3	12.7	18.2	30.8	30.8	14.4	10.9	15.5	19.3	12.7	22.2	16	15.8
Llama-3.1-8B-Instruct2025.12	25	21.1	28.6	70	26.7	24.2	18.9	42.1	22.9	57.1	12.5	36.4	30.8	38.5	26.8	28.6	32.4	27	13.6	33.3	24.1	26
Qwen3-0.6B2025.12	0	15.8	18.2	44.4	10.2	18.2	16.2	21.1	5.7	0	9.7	17.6	16.7	15.4	18.9	18.5	8.6	14	7.5	22.2	14.5	13.4
Llama-3.2-1B-Instruct2025.12	0	15.8	14.3	0	6.7	6.1	5.4	27	5.7	14.3	9.7	12.1	32	24	10.2	14.5	11.6	12.4	12.6	11.1	10.2	11.5