Code Question Answering on CodeSimpleQA Chinese
100ADSQwen3-30B-A3B-Thinking-2507
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-30B-A3B-Thinking-2507Params=3B/30B2025.12 | 100 | 68.4 | 42.9 | 80 | 46.7 | 47.1 | 51.4 | 63.2 | 51.4 | 57.1 | 31.2 | 64.7 | 46.2 | 53.8 | 45.4 | 39.3 | 40.8 | 47.5 | 42.2 | 58.8 | 40.2 | 46.7 | |
| Qwen2.5-1.5B-InstructParams=1.5B2025.12 | 85.7 | 26.3 | 14.3 | 40 | 23.7 | 12.1 | 21.6 | 47.4 | 14.5 | 28.6 | 6.3 | 24.2 | 15.4 | 53.8 | 20.4 | 21.8 | 21.1 | 16.9 | 13.8 | 22.2 | 16.3 | 20 | |
| gpt-4o-2024-11-202025.12 | 75 | 78.6 | 60 | 87.5 | 50 | 41.7 | 51 | 73.3 | 44 | 75 | 32 | 38.5 | 60 | 55.6 | 46.6 | 50 | 55.1 | 53.7 | 48 | 40 | 43.2 | 50.7 | |
| o32025.12 | 75 | 68.4 | 71.4 | 80 | 70 | 58.8 | 67.6 | 68.4 | 61.8 | 57.1 | 56.2 | 58.8 | 46.2 | 61.5 | 63.3 | 57.1 | 73.2 | 59.1 | 57.3 | 66.7 | 57 | 62.1 | |
| claude-sonnet-4-20250514-thinking2025.12 | 75 | 73.3 | 57.1 | 100 | 53.8 | 47.6 | 58.1 | 80 | 54.2 | 44.4 | 51.2 | 31.6 | 36.4 | 70 | 59.8 | 60 | 64.5 | 46.2 | 50.9 | 76.9 | 47.1 | 55.4 | |
| gpt-52025.12 | 75 | 90.9 | 80 | 100 | 77.8 | 62.5 | 75.5 | 84.6 | 60 | 50 | 57.1 | 50 | 66.7 | 55.6 | 74.2 | 62.1 | 88.4 | 58.1 | 67.4 | 80 | 47.5 | 67.2 | |
| DeepSeek-R1-Distill-Qwen-1.5BParams=1.5B2025.12 | 75 | 16.7 | 0 | 31.6 | 6.9 | 6.2 | 5.8 | 17.6 | 3 | 14.3 | 3.6 | 11.8 | 7.7 | 30.8 | 2.3 | 10.9 | 0 | 3.8 | 2.6 | 11.8 | 3.2 | 6.2 | |
| Qwen3-4B-Thinking-2507Params=4B2025.12 | 75 | 78.9 | 14.3 | 70 | 43.3 | 35.3 | 41.7 | 47.4 | 45.7 | 42.9 | 25 | 42.4 | 32 | 38.5 | 30.6 | 35.7 | 39.4 | 31.6 | 29.6 | 66.7 | 39.2 | 38.3 | |
| Qwen2.5-7B-InstructParams=7B2025.12 | 75 | 48.6 | 28.6 | 42.1 | 23.3 | 17.6 | 27 | 36.8 | 22.9 | 42.9 | 15.6 | 30.3 | 15.4 | 46.2 | 27.1 | 36.4 | 28.4 | 24 | 20.9 | 22.2 | 22.3 | 26.3 | |
| Seed-Coder-8B-InstructParams=8B2025.12 | 75 | 21.1 | 14.3 | 30 | 40 | 24.2 | 21.6 | 32.4 | 22.9 | 42.9 | 9.4 | 35.3 | 16 | 46.2 | 35.4 | 35.7 | 32.9 | 32.6 | 26.3 | 33.3 | 31.5 | 29.8 | |
| DeepSeek-R1-0528-Qwen3-8BParams=8B2025.12 | 75 | 32.4 | 30.8 | 21.1 | 38.5 | 24.2 | 36.1 | 48.5 | 26.9 | 50 | 32.3 | 44.4 | 33.3 | 24 | 29.2 | 34.6 | 32.1 | 27.2 | 21.3 | 35.3 | 24.6 | 30 | |
| Qwen2.5-14B-InstructParams=14B2025.12 | 75 | 47.4 | 28.6 | 50 | 37.9 | 29.4 | 37.8 | 52.6 | 43.5 | 42.9 | 12.9 | 41.2 | 15.4 | 30.8 | 31.2 | 32.7 | 38 | 33.9 | 23.2 | 55.6 | 23.1 | 32.5 | |
| gpt-oss-20bParams=3.6B/21B2025.12 | 75 | 47.4 | 42.9 | 70 | 54.2 | 35.3 | 43.2 | 47.4 | 43.5 | 42.9 | 25.4 | 42.4 | 46.2 | 38.5 | 43.3 | 39.3 | 49.6 | 40.7 | 30.9 | 55.6 | 50 | 43.3 | |
| Qwen2.5-32B-InstructParams=32B2025.12 | 75 | 52.6 | 42.9 | 60 | 36.7 | 29.4 | 37.8 | 47.4 | 17.1 | 14.3 | 25 | 35.3 | 32 | 46.2 | 39.2 | 39.3 | 45.1 | 35 | 29.4 | 35.3 | 39.4 | 36.7 | |
| DeepSeek-R1-Distill-Qwen-32BParams=32B2025.12 | 75 | 42.1 | 14.3 | 30 | 30 | 17.6 | 29.7 | 59.5 | 31.9 | 28.6 | 21.9 | 25 | 23.1 | 48 | 41.2 | 28.6 | 34 | 30.1 | 29.6 | 47.1 | 35.4 | 32.8 | |
| QwQ-32BParams=32B2025.12 | 75 | 63.2 | 30.8 | 40 | 40 | 41.2 | 41.7 | 75.7 | 50 | 71.4 | 36.1 | 41.2 | 64 | 61.5 | 45.8 | 32.7 | 51.1 | 35.8 | 33.8 | 47.1 | 43.7 | 44 | |
| Seed-OSSParams=36B2025.12 | 75 | 52.6 | 0 | 50 | 36.7 | 36.4 | 43.8 | 52.6 | 48.6 | 42.9 | 16.1 | 36.4 | 32 | 46.2 | 37.5 | 32.1 | 39.7 | 32.8 | 27.5 | 47.1 | 38 | 36.8 | |
| DeepSeek-R1-Distill-Llama-70BParams=70B2025.12 | 75 | 48.6 | 15.4 | 30 | 36.7 | 29.4 | 32.4 | 47.4 | 49.3 | 28.6 | 25.4 | 41.2 | 46.2 | 38.5 | 41.7 | 28.6 | 46.5 | 34.7 | 28 | 35.3 | 44 | 38 | |
| Qwen3-235B-A22B-Instruct-2507Params=22B/235B2025.12 | 75 | 68.4 | 42.9 | 70 | 53.3 | 35.3 | 56.8 | 78.9 | 51.4 | 57.1 | 28.1 | 50 | 46.2 | 53.8 | 49 | 60.7 | 56.3 | 48.3 | 43.9 | 55.6 | 42 | 49.9 | |
| Qwen3-235B-A22B-Thinking-2507Params=22B/235B2025.12 | 75 | 78.9 | 28.6 | 90 | 53.3 | 47.1 | 62.2 | 68.4 | 43.5 | 71.4 | 37.5 | 58.8 | 53.8 | 46.2 | 53.1 | 50 | 57.7 | 50.8 | 40.5 | 44.4 | 56.3 | 52.9 | |
| Qwen3-Coder-480B-A35B-InstructParams=35B/480B2025.12 | 75 | 68.4 | 42.9 | 70 | 50 | 36.4 | 48.6 | 64.9 | 37.7 | 57.1 | 34.4 | 35.3 | 61.5 | 61.5 | 49.5 | 47.3 | 60.6 | 45.2 | 42 | 55.6 | 40 | 47.7 | |
| DeepSeek-V3Params=37B/671B2025.12 | 75 | 57.9 | 42.9 | 80 | 46.7 | 52.9 | 51.4 | 73.7 | 48.6 | 71.4 | 40.6 | 54.5 | 46.2 | 61.5 | 44.9 | 46.4 | 57.7 | 46.1 | 40.2 | 44.4 | 48.2 | 49.6 | |
| DeepSeek-V3-0324Params=37B/671B2025.12 | 75 | 84.2 | 57.1 | 90 | 60 | 47.1 | 56.8 | 63.2 | 54.3 | 71.4 | 34.4 | 41.2 | 38.5 | 53.8 | 53.6 | 42.9 | 59.2 | 46.1 | 45.1 | 55.6 | 47 | 51.6 | |
| DeepSeek-V3.1Params=37B/671B2025.12 | 75 | 78.9 | 57.1 | 80 | 56.7 | 29.4 | 54.1 | 68.4 | 54.3 | 42.9 | 21.9 | 47.1 | 38.5 | 61.5 | 44.9 | 57.1 | 60.6 | 47.7 | 39 | 66.7 | 46.2 | 49.8 | |
| claude-sonnet-4-202505142025.12 | 66.7 | 61.5 | 28.6 | 85.7 | 60 | 51.9 | 51.9 | 53.3 | 54.5 | 36.4 | 38.1 | 33.3 | 33.3 | 42.9 | 58.3 | 70.3 | 64.3 | 54.7 | 44.1 | 76.9 | 49.4 | 53.5 | |
| Qwen3-8BParams=8B2025.12 | 57.1 | 57.9 | 46.2 | 52.6 | 37.3 | 29.4 | 43.2 | 57.9 | 45.7 | 85.7 | 28.6 | 29.4 | 15.4 | 38.5 | 31.2 | 35.7 | 44 | 33 | 30.9 | 47.1 | 29.7 | 36.8 | |
| chatgpt-4o-latest2025.12 | 50 | 73.7 | 46.2 | 70 | 63.3 | 54.5 | 51.4 | 78.9 | 51.4 | 42.9 | 43.8 | 52.9 | 46.2 | 53.8 | 49 | 50 | 59.2 | 49.4 | 45.1 | 55.6 | 42 | 51.4 | |
| o3-mini-2025-01-312025.12 | 50 | 68.4 | 42.9 | 100 | 66.7 | 47.1 | 45.9 | 73.7 | 51.4 | 85.7 | 34.9 | 47.1 | 53.8 | 53.8 | 61.9 | 50 | 57.7 | 47.2 | 47.9 | 47.1 | 52.3 | 53.4 | |
| o4-mini2025.12 | 50 | 72.2 | 50 | 100 | 63.3 | 46.7 | 54.3 | 84.2 | 58.8 | 83.3 | 41.4 | 60 | 50 | 53.8 | 56.5 | 44.4 | 70.6 | 46.1 | 40 | 77.8 | 54.6 | 55.5 | |
| Qwen2.5-Coder-1.5B-InstructParams=1.5B2025.12 | 50 | 16.2 | 28.6 | 21.1 | 16.9 | 5.9 | 10.8 | 27 | 17.1 | 14.3 | 9.4 | 18.2 | 23.1 | 30.8 | 16.5 | 25 | 25.5 | 19.2 | 12.7 | 11.1 | 16.5 | 17.9 | |
| Qwen2.5-Coder-3B-InstructParams=3B2025.12 | 50 | 31.6 | 42.9 | 11.1 | 27.1 | 11.8 | 30.1 | 36.8 | 23.5 | 28.6 | 9.7 | 36.4 | 23.1 | 61.5 | 28.9 | 25 | 25.9 | 22.6 | 19 | 35.3 | 25.6 | 25.5 | |
| Qwen2.5-3B-InstructParams=3B2025.12 | 50 | 16.2 | 14.3 | 40 | 13.6 | 12.1 | 24.7 | 36.8 | 14.3 | 14.3 | 9.4 | 42.4 | 24 | 46.2 | 14.4 | 32.7 | 17 | 20.7 | 16.2 | 23.5 | 18.1 | 20 | |
| Qwen3-4BParams=4B2025.12 | 50 | 54.1 | 28.6 | 63.2 | 40 | 24.2 | 35.6 | 42.1 | 31.9 | 57.1 | 25 | 23.5 | 30.8 | 53.8 | 33 | 28.6 | 36.5 | 31 | 31.2 | 50 | 33.5 | 34.5 | |
| DeepSeek-R1-Distill-Qwen-7BParams=7B2025.12 | 50 | 31.6 | 14.3 | 30 | 13.3 | 11.8 | 16.4 | 31.6 | 8.6 | 0 | 9.5 | 12.1 | 8 | 32 | 6.2 | 7.3 | 10.1 | 12.6 | 5 | 44.4 | 5 | 11.6 | |
| Qwen2.5-Coder-7B-InstructParams=7B2025.12 | 50 | 31.6 | 28.6 | 40 | 30.5 | 17.6 | 16.4 | 36.8 | 20.3 | 42.9 | 15.6 | 41.2 | 15.4 | 23.1 | 26.8 | 25 | 34.5 | 28.2 | 25.9 | 11.1 | 31.5 | 27.6 | |
| Qwen3-1.7BParams=7B2025.12 | 50 | 52.6 | 14.3 | 50 | 36.7 | 17.6 | 19.2 | 57.9 | 23.2 | 42.9 | 21.9 | 17.6 | 24 | 48 | 26.5 | 18.2 | 21.4 | 13.5 | 18.3 | 47.1 | 16 | 23.4 | |
| Hunyuan-7B-InstructParams=7B2025.12 | 50 | 31.6 | 28.6 | 40 | 26.7 | 12.1 | 29.7 | 21.6 | 14.3 | 14.3 | 19 | 11.8 | 23.1 | 38.5 | 14.4 | 25 | 23 | 25 | 13.7 | 11.1 | 18.3 | 21 | |
| DeepSeek-R1-Distill-Llama-8BParams=8B2025.12 | 50 | 21.1 | 14.3 | 31.6 | 16.7 | 17.6 | 8.7 | 31.6 | 11.9 | 28.6 | 3.2 | 30.3 | 33.3 | 38.5 | 14.9 | 10.9 | 18.4 | 16.2 | 12.8 | 22.2 | 15.4 | 16.7 | |
| Qwen2.5-Coder-14B-InstructParams=14B2025.12 | 50 | 47.4 | 42.9 | 31.6 | 30 | 42.4 | 29.7 | 36.8 | 28.6 | 57.1 | 19 | 41.2 | 23.1 | 46.2 | 38.8 | 46.4 | 42.3 | 31.5 | 32.1 | 33.3 | 30.2 | 34.5 | |
| DeepSeek-R1-Distill-Qwen-14BParams=14B2025.12 | 50 | 48.6 | 28.6 | 50 | 30 | 32.3 | 30.1 | 42.1 | 31.9 | 42.9 | 9.4 | 24.2 | 15.4 | 61.5 | 39.6 | 25.5 | 35.7 | 32.6 | 25.8 | 35.3 | 29.1 | 31.6 | |
| Qwen3-30B-A3B-Instruct-2507Params=3B/30B2025.12 | 50 | 68.4 | 42.9 | 60 | 36.7 | 35.3 | 51.4 | 63.2 | 34.3 | 57.1 | 21.9 | 41.2 | 38.5 | 46.2 | 42.9 | 53.6 | 45.1 | 39.5 | 37.8 | 44.4 | 40 | 42.3 | |
| Qwen3-30B-A3BParams=3B/30B2025.12 | 50 | 42.1 | 57.1 | 60 | 43.3 | 35.3 | 49.3 | 57.9 | 42.9 | 57.1 | 25 | 29.4 | 46.2 | 53.8 | 46.3 | 50 | 40.8 | 33.3 | 32.5 | 35.3 | 43.4 | 41 | |
| Qwen3-Coder-30B-A3B-InstructParams=3B/30B2025.12 | 50 | 68.4 | 57.1 | 50 | 33.3 | 37.5 | 45.9 | 63.2 | 37.1 | 57.1 | 28.1 | 23.5 | 53.8 | 53.8 | 24.5 | 42.9 | 45.1 | 30.3 | 33.1 | 55.6 | 36 | 38.5 | |
| Qwen3-32BParams=32B2025.12 | 50 | 57.9 | 28.6 | 70 | 36.7 | 29.4 | 49.3 | 63.2 | 51.4 | 71.4 | 28.1 | 47.1 | 38.5 | 53.8 | 46.9 | 47.3 | 56.7 | 37.5 | 39.8 | 44.4 | 36.5 | 44.1 | |
| Qwen2.5-72B-InstructParams=72B2025.12 | 50 | 63.2 | 30.8 | 50 | 30 | 29.4 | 43.2 | 63.2 | 42.9 | 42.9 | 21.9 | 42.4 | 23.1 | 53.8 | 41.2 | 50 | 45.7 | 40.7 | 30.7 | 44.4 | 41.4 | 40.5 | |
| gpt-oss-120bParams=5.1B/117B2025.12 | 50 | 73.7 | 42.9 | 70 | 63.3 | 41.2 | 54.1 | 63.2 | 62.9 | 85.7 | 46.9 | 47.1 | 38.5 | 53.8 | 57.1 | 50 | 62 | 50.6 | 46.3 | 55.6 | 51 | 54.1 | |
| Qwen3-235B-A22BParams=22B/235B2025.12 | 50 | 52.6 | 42.9 | 80 | 43.3 | 35.3 | 51.4 | 68.4 | 48.6 | 71.4 | 31.2 | 36.4 | 38.5 | 38.5 | 49.5 | 39.3 | 57.7 | 42 | 31.9 | 55.6 | 39 | 44.5 | |
| DeepSeek-R1Params=37B/671B2025.12 | 50 | 63.2 | 28.6 | 60 | 43.3 | 35.3 | 48.6 | 57.9 | 57.1 | 71.4 | 37.5 | 41.2 | 46.2 | 30.8 | 52.1 | 50 | 52.1 | 50.8 | 41.5 | 66.7 | 46 | 48.2 | |
| DeepSeek-R1-0528Params=37B/671B2025.12 | 50 | 47.4 | 28.6 | 70 | 50 | 35.3 | 54.1 | 68.4 | 51.4 | 57.1 | 25 | 52.9 | 46.2 | 69.2 | 53.1 | 39.3 | 49.3 | 47.2 | 39.3 | 44.4 | 48.2 | 47.5 | |
| GLM-4.5Params=32B/355B2025.12 | 50 | 63.2 | 42.9 | 90 | 53.3 | 41.2 | 51.4 | 63.2 | 54.3 | 71.4 | 25 | 41.2 | 53.8 | 46.2 | 51 | 46.4 | 56.7 | 52 | 47.9 | 58.8 | 49.2 | 50.9 | |
| GLM-4.5-AirParams=12B/106B2025.12 | 50 | 63.2 | 42.9 | 90 | 46.7 | 5.9 | 48.6 | 52.6 | 45.7 | 57.1 | 28.1 | 47.1 | 38.5 | 38.5 | 49 | 29.1 | 56.3 | 46.1 | 34.1 | 58.8 | 42.4 | 44.3 | |
| Qwen2.5-Coder-32B-InstructParams=32B2025.12 | 50 | 63.2 | 28.6 | 52.6 | 40 | 36.4 | 40.5 | 68.4 | 34.8 | 42.9 | 28.1 | 42.4 | 15.4 | 53.8 | 43.3 | 42.9 | 44 | 38.2 | 35.4 | 55.6 | 35 | 40 | |
| CodeSimpleQA-RFTParams=32B2025.12 | 50 | 70.6 | 50 | 41.7 | 46.7 | 41.2 | 38.5 | 52.4 | 42.9 | 28.6 | 27.3 | 41.2 | 20 | 53.8 | 31.4 | 64.3 | 48.6 | 39.3 | 38.4 | 55.6 | 42.2 | 42.3 | |
| CodeSimpleQA-RLParams=32B2025.12 | 50 | 78.8 | 37.5 | 60.9 | 43.3 | 38.7 | 39 | 57.1 | 48.6 | 28.6 | 37.5 | 52.9 | 33.3 | 61.5 | 35.6 | 43.6 | 53.1 | 45.9 | 39 | 55.6 | 43.5 | 45.2 | |
| gpt-5-mini2025.12 | 40 | 78.6 | 100 | 85.7 | 58.5 | 50 | 58.3 | 69.6 | 57.8 | 100 | 44.4 | 71 | 55.6 | 66.7 | 68.6 | 46.7 | 66.7 | 45.8 | 52.7 | 0 | 54 | 57.8 | |
| Qwen3-14BParams=14B2025.12 | 28.6 | 68.4 | 42.9 | 70 | 36.7 | 35.3 | 45.9 | 47.4 | 31.4 | 85.7 | 31.2 | 36.4 | 61.5 | 30.8 | 42.9 | 44.4 | 46.8 | 35.6 | 36 | 55.6 | 33.2 | 40.5 | |
| Qwen2.5-0.5B-InstructParams=0.5B2025.12 | 25 | 11.8 | 15.4 | 0 | 13.8 | 0 | 24.7 | 21.6 | 5.7 | 30.8 | 12.9 | 24.2 | 0 | 24 | 12.5 | 25.5 | 12.8 | 12.9 | 10.1 | 22.2 | 11.3 | 13.5 | |
| Qwen2.5-Coder-0.5B-InstructParams=0.5B2025.12 | 25 | 5.4 | 0 | 11.1 | 14.5 | 6.5 | 8.5 | 21.1 | 2.9 | 0 | 9.8 | 13.3 | 16.7 | 30.8 | 14.9 | 7.3 | 14.3 | 14 | 16.6 | 0 | 15.1 | 13 | |
| Llama-3.2-3B-InstructParams=3B2025.12 | 25 | 21.1 | 14.3 | 10 | 16.9 | 11.8 | 10.8 | 15.8 | 14.3 | 14.3 | 12.7 | 18.2 | 30.8 | 30.8 | 14.4 | 10.9 | 15.5 | 19.3 | 12.7 | 22.2 | 16 | 15.8 | |
| Llama-3.1-8B-InstructParams=8B2025.12 | 25 | 21.1 | 28.6 | 70 | 26.7 | 24.2 | 18.9 | 42.1 | 22.9 | 57.1 | 12.5 | 36.4 | 30.8 | 38.5 | 26.8 | 28.6 | 32.4 | 27 | 13.6 | 33.3 | 24.1 | 26 | |
| Qwen3-0.6BParams=0.6B2025.12 | 0 | 15.8 | 18.2 | 44.4 | 10.2 | 18.2 | 16.2 | 21.1 | 5.7 | 0 | 9.7 | 17.6 | 16.7 | 15.4 | 18.9 | 18.5 | 8.6 | 14 | 7.5 | 22.2 | 14.5 | 13.4 | |
| Llama-3.2-1B-InstructParams=1B2025.12 | 0 | 15.8 | 14.3 | 0 | 6.7 | 6.1 | 5.4 | 27 | 5.7 | 14.3 | 9.7 | 12.1 | 32 | 24 | 10.2 | 14.5 | 11.6 | 12.4 | 12.6 | 11.1 | 10.2 | 11.5 |