Factual code question answering on CodeSimpleQA English 1.0
60ADSchatgpt-4o-latest
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| chatgpt-4o-latestParams=Proprietary2025.12 | 60 | 50 | 0 | 63.6 | 58.8 | 30 | 62.5 | 40.6 | 48.3 | 75 | 66.7 | 59.3 | 23.1 | 25 | 63.9 | 45 | 43.1 | 58.1 | 36.4 | 0 | 51.4 | 50.5 | |
| gpt-5-miniParams=Proprietary2025.12 | 50 | 60 | 50 | 70.6 | 53.8 | 57.1 | 47.1 | 50 | 40.5 | 50 | 54.5 | 75 | 41.7 | 25 | 68.2 | 50 | 53.7 | 58.4 | 46.7 | 100 | 56.5 | 53.5 | |
| o3-mini-2025-01-31Params=Proprietary2025.12 | 50 | 66.7 | 50 | 66.7 | 52.9 | 40 | 54.2 | 50 | 48.3 | 58.3 | 57.7 | 50 | 30.8 | 62.5 | 58.3 | 40 | 35.3 | 51.4 | 37.2 | 100 | 45.2 | 47.5 | |
| gpt-oss-20bParams=3.6B/21B2025.12 | 50 | 16.7 | 28.6 | 27.3 | 17.6 | 20 | 38.3 | 28.1 | 36.2 | 43.5 | 57.7 | 28.6 | 23.1 | 37.5 | 47.2 | 30 | 39.6 | 36.7 | 27.8 | 50 | 38 | 35.8 | |
| Qwen3-235B-A22BParams=235B2025.12 | 50 | 50 | 25 | 63.6 | 29.4 | 40 | 29.2 | 50.8 | 36.2 | 58.3 | 50 | 42.9 | 23.1 | 25 | 45.1 | 55 | 39.2 | 49.1 | 31 | 100 | 40.8 | 42 | |
| Qwen3-235B-A22B-Thinking-2507Params=235B2025.12 | 50 | 50 | 0 | 54.5 | 58.8 | 40 | 50 | 34.4 | 43.1 | 41.7 | 61.5 | 50 | 30.8 | 25 | 52.8 | 30 | 25.5 | 50.2 | 32.1 | 50 | 41.7 | 42.5 | |
| Qwen3-Coder-480B-A35B-InstructParams=480B2025.12 | 50 | 33.3 | 0 | 28.6 | 52.9 | 30 | 46.8 | 19.4 | 40 | 58.3 | 39.1 | 64.3 | 48 | 40 | 70.4 | 45 | 47.1 | 44.8 | 28 | 100 | 43.7 | 42.6 | |
| claude-sonnet-4-20250514-thinkingParams=Proprietary2025.12 | 44.4 | 25 | 50 | 50 | 36.4 | 61.5 | 58.3 | 54.5 | 40.4 | 35.3 | 44.4 | 50 | 36.4 | 46.2 | 45.3 | 62.5 | 49.3 | 55.8 | 43 | 100 | 53.2 | 49.6 | |
| GLM-4.5-AirParams=12B/102B2025.12 | 44.4 | 36.4 | 0 | 57.1 | 24.2 | 40 | 31.1 | 39.3 | 33 | 26.1 | 34 | 51.9 | 24 | 13.3 | 58 | 30.8 | 38.3 | 37.5 | 26.2 | 100 | 39.5 | 36.5 | |
| Qwen3-Coder-30B-A3B-InstructParams=30B2025.12 | 42.1 | 50 | 0 | 38.1 | 35.3 | 20 | 38.3 | 34.9 | 29.6 | 8.7 | 29.2 | 57.1 | 16 | 0 | 39.4 | 30 | 33.3 | 33.8 | 23.9 | 100 | 30.4 | 31.3 | |
| Qwen3-235B-A22B-Instruct-2507Params=235B2025.12 | 42.1 | 50 | 0 | 54.5 | 47.1 | 40 | 41.7 | 37.5 | 44.8 | 58.3 | 42.3 | 35.7 | 30.8 | 37.5 | 52.8 | 40 | 35.3 | 43.3 | 34 | 100 | 40.6 | 41.2 | |
| o3Params=Proprietary2025.12 | 40 | 33.3 | 50 | 72.7 | 82.4 | 60 | 70.8 | 59.4 | 69 | 66.7 | 57.7 | 78.6 | 30.8 | 62.5 | 72.2 | 65 | 60.8 | 62.5 | 54.3 | 100 | 58 | 61.2 | |
| o4-miniParams=Proprietary2025.12 | 40 | 50 | 25 | 60 | 58.8 | 30 | 50 | 56.7 | 57.9 | 50 | 68 | 42.9 | 38.5 | 50 | 65.7 | 40 | 51.1 | 55.8 | 39.3 | 100 | 47.9 | 51.1 | |
| Qwen2.5-Coder-7B-InstructParams=7B2025.12 | 40 | 16.7 | 0 | 9.1 | 17.6 | 20 | 42.6 | 21.9 | 24.1 | 16.7 | 34.6 | 29.6 | 8 | 12.5 | 44.4 | 25 | 26 | 29.5 | 15 | 100 | 23.2 | 25.2 | |
| Seed-Coder-8B-InstructParams=8B2025.12 | 40 | 18.2 | 0 | 18.2 | 11.8 | 10 | 37.5 | 12.5 | 25.9 | 25 | 26.9 | 29.6 | 0 | 0 | 33.3 | 30 | 23.8 | 26.5 | 10.8 | 0 | 29.6 | 23.8 | |
| Qwen3-8BParams=8B2025.12 | 40 | 36.4 | 0 | 18.2 | 23.5 | 10 | 33.3 | 29 | 22.6 | 34.8 | 28.6 | 28.6 | 0 | 0 | 37.1 | 45 | 29.7 | 32.7 | 9.8 | 50 | 30.3 | 27.1 | |
| Qwen2.5-Coder-14B-InstructParams=14B2025.12 | 40 | 16.7 | 0 | 36.4 | 29.4 | 10 | 45.8 | 21.9 | 24.1 | 50 | 19.2 | 51.9 | 15.4 | 12.5 | 50 | 30 | 30 | 32.8 | 19.3 | 50 | 32.1 | 30 | |
| Qwen2.5-14B-InstructParams=14B2025.12 | 40 | 0 | 0 | 27.3 | 23.5 | 30 | 33.3 | 28.1 | 27.6 | 33.3 | 42.3 | 29.6 | 7.7 | 25 | 38.9 | 20.5 | 25.7 | 27.8 | 12.9 | 100 | 30.6 | 27.2 | |
| Qwen3-14BParams=14B2025.12 | 40 | 36.4 | 25 | 36.4 | 23.5 | 10 | 41.7 | 31.7 | 26.3 | 41.7 | 52 | 35.7 | 7.7 | 0 | 50.7 | 50 | 23.5 | 31.1 | 21.6 | 50 | 31.1 | 30.9 | |
| Qwen3-30B-A3B-Instruct-2507Params=30B2025.12 | 40 | 33.3 | 0 | 36.4 | 29.4 | 30 | 45.8 | 25 | 34.5 | 58.3 | 42.3 | 50 | 38.5 | 12.5 | 52.8 | 35 | 31.4 | 36.9 | 17 | 100 | 37.1 | 34.7 | |
| Qwen3-30B-A3BParams=30B2025.12 | 40 | 33.3 | 0 | 45.5 | 23.5 | 20 | 37.5 | 22.6 | 31.3 | 41.7 | 43.1 | 64.3 | 23.1 | 0 | 56.3 | 35 | 37.3 | 33.7 | 16.3 | 100 | 33.5 | 32.7 | |
| QwQ-32BParams=32B2025.12 | 40 | 36.4 | 25 | 63.6 | 29.4 | 30 | 41.7 | 34.9 | 28.6 | 17.4 | 34.6 | 42.9 | 15.4 | 0 | 53.5 | 30 | 29.7 | 36.2 | 22.6 | 50 | 39.8 | 34.4 | |
| Qwen2.5-72B-InstructParams=72B2025.12 | 40 | 33.3 | 25 | 45.5 | 41.2 | 10 | 45.8 | 34.4 | 37.9 | 41.7 | 46.2 | 37 | 7.7 | 50 | 52.8 | 25 | 37.6 | 44.9 | 28.9 | 50 | 40.1 | 38.9 | |
| gpt-oss-120bParams=5.1B/117B2025.12 | 40 | 50 | 0 | 54.5 | 29.4 | 40 | 45.8 | 53.1 | 50 | 41.7 | 61.5 | 64.3 | 23.1 | 37.5 | 52.8 | 35 | 52.9 | 51.2 | 34 | 100 | 46.2 | 46.6 | |
| DeepSeek-R1Params=37B/671B2025.12 | 40 | 33.3 | 50 | 54.5 | 35.3 | 30 | 54.2 | 50 | 41.4 | 41.7 | 61.5 | 42.9 | 38.5 | 12.5 | 47.2 | 55 | 51 | 53.5 | 31.9 | 100 | 46.2 | 46.2 | |
| DeepSeek-R1-0528Params=37B/671B2025.12 | 40 | 50 | 25 | 54.5 | 35.3 | 40 | 58.3 | 50 | 36.2 | 50 | 53.8 | 57.1 | 46.2 | 37.5 | 61.1 | 40 | 37.3 | 50.6 | 32.4 | 100 | 42.1 | 44.5 | |
| DeepSeek-V3Params=37B/671B2025.12 | 40 | 33.3 | 25 | 54.5 | 29.4 | 20 | 58.3 | 46.9 | 43.1 | 66.7 | 46.2 | 50 | 30.8 | 12.5 | 52.8 | 40 | 33.3 | 45.6 | 26.6 | 50 | 40 | 40.7 | |
| DeepSeek-V3-0324Params=37B/671B2025.12 | 40 | 33.3 | 50 | 63.6 | 41.2 | 40 | 54.2 | 50 | 36.2 | 66.7 | 50 | 64.3 | 23.1 | 12.5 | 50 | 50 | 37.3 | 44.4 | 35.1 | 50 | 44.5 | 43.4 | |
| GLM-4.5Params=32B/355B2025.12 | 40 | 66.7 | 25 | 66.7 | 29.4 | 40 | 45.8 | 66.7 | 43.1 | 50 | 57.7 | 50 | 30.8 | 12.5 | 47.2 | 50 | 35.6 | 47.5 | 41.9 | 50 | 43.1 | 45 | |
| Qwen2.5-Coder-32B-InstructParams=32B2025.12 | 40 | 16.7 | 0 | 54.5 | 47.1 | 10 | 37.5 | 28.1 | 31.3 | 33.3 | 39.2 | 51.9 | 7.7 | 25 | 52.8 | 25 | 29.4 | 37.5 | 26.6 | 100 | 33.5 | 33.9 | |
| CodeSimpleQA-RFTParams=32B2025.12 | 36.4 | 25 | 0 | 63.6 | 41.2 | 20 | 33.3 | 34.4 | 34.5 | 16.7 | 53.8 | 51.9 | 15.4 | 25 | 66.7 | 25 | 33.7 | 41.6 | 34.2 | 50 | 41.7 | 39.2 | |
| gpt-5Params=Proprietary2025.12 | 35.3 | 20 | 100 | 66.7 | 69.2 | 80 | 62.9 | 78.8 | 48.8 | 81.8 | 57.1 | 100 | 40 | 71.4 | 65.2 | 66.7 | 54.3 | 68.4 | 58 | 100 | 64.2 | 62.9 | |
| claude-sonnet-4-20250514Params=Proprietary2025.12 | 33.3 | 66.7 | 50 | 35.3 | 44.4 | 36.4 | 61.5 | 57.1 | 50 | 55.6 | 50 | 63.6 | 16.7 | 61.5 | 68.1 | 62.5 | 51.9 | 53.3 | 37.1 | 0 | 51.6 | 50.3 | |
| Llama-3.1-8B-InstructParams=8B2025.12 | 31.6 | 33.3 | 25 | 38.1 | 23.5 | 10 | 36.4 | 16.1 | 23.2 | 27.3 | 8.9 | 21.4 | 7.7 | 40 | 50.7 | 5.1 | 32 | 29.6 | 14.6 | 100 | 21 | 24.4 | |
| Qwen3-4B-Thinking-2507Params=4B2025.12 | 30 | 33.3 | 25 | 27.3 | 23.5 | 20 | 25 | 22.2 | 24.6 | 16.7 | 38.5 | 28.6 | 8 | 0 | 31 | 20 | 21.6 | 31.1 | 12.9 | 50 | 25.7 | 24.7 | |
| Qwen3-30B-A3B-Thinking-2507Params=30B2025.12 | 30 | 16.7 | 25 | 36.4 | 23.5 | 20 | 34 | 28.6 | 27.6 | 41.7 | 43.1 | 50 | 16 | 0 | 44.4 | 35 | 31.7 | 32.2 | 16 | 50 | 35.4 | 30.9 | |
| Qwen2.5-32B-InstructParams=32B2025.12 | 30 | 16.7 | 25 | 47.6 | 41.2 | 30 | 29.2 | 28.6 | 34.5 | 26.1 | 42.3 | 28.6 | 0 | 25 | 50 | 15 | 30 | 39.2 | 17.2 | 100 | 33.6 | 32.3 | |
| DeepSeek-R1-Distill-Qwen-32BParams=32B2025.12 | 30 | 33.3 | 0 | 36.4 | 23.5 | 20 | 33.3 | 12.5 | 24.1 | 26.1 | 34.6 | 21.4 | 0 | 0 | 52.8 | 15.4 | 31.4 | 29.5 | 19.1 | 50 | 30.1 | 27.2 | |
| Qwen3-32BParams=32B2025.12 | 30 | 33.3 | 25 | 57.1 | 29.4 | 20 | 42.6 | 50 | 24.3 | 41.7 | 47.1 | 35.7 | 15.4 | 37.5 | 37.1 | 25.6 | 31.7 | 45.9 | 19.4 | 50 | 37 | 35.6 | |
| Seed-OSSParams=36B2025.12 | 30 | 16.7 | 25 | 27.3 | 29.4 | 20 | 33.3 | 34.9 | 26.1 | 16.7 | 29.2 | 28.6 | 24 | 0 | 42.3 | 30 | 35.3 | 33.1 | 25.7 | 50 | 32.7 | 30.7 | |
| Qwen3-Coder-480BParams=480B2025.12 | 30 | 18.2 | 0 | 11.1 | 43.7 | 27.3 | 42.9 | 13.8 | 35.5 | 60.9 | 34.8 | 64.3 | 34.8 | 42.9 | 64.5 | 32.4 | 33.1 | 39.9 | 32.6 | 100 | 42.4 | 38.2 | |
| DeepSeek-V3.1Params=37B/671B2025.12 | 30 | 0 | 50 | 54.5 | 41.2 | 30 | 45.8 | 46.9 | 43.1 | 50 | 38.5 | 50 | 15.4 | 25 | 47.2 | 20 | 41.6 | 49.4 | 30.9 | 50 | 38.2 | 40.5 | |
| gpt-4o-2024-11-20Params=Proprietary2025.12 | 28.6 | 50 | 33.3 | 73.7 | 50 | 33.3 | 62.5 | 42.3 | 39.5 | 66.7 | 68.4 | 54.5 | 20 | 0 | 68 | 35.3 | 44.4 | 51.5 | 36.9 | 50 | 45.5 | 46.7 | |
| CodeSimpleQA-RLParams=32B2025.12 | 28.6 | 50 | 0 | 63.6 | 35.3 | 20 | 37.5 | 31.7 | 43.1 | 33.3 | 58.8 | 50 | 40 | 26.7 | 62 | 45 | 37.6 | 46.5 | 29.9 | 100 | 44.6 | 42.3 | |
| Qwen2.5-0.5B-InstructParams=0.5B2025.12 | 21.1 | 0 | 0 | 36.4 | 12.1 | 0 | 8.7 | 3.2 | 1.8 | 8.7 | 3.9 | 7.4 | 0 | 26.7 | 22.9 | 5.1 | 12 | 15.5 | 1.1 | 50 | 14.9 | 11 | |
| Qwen2.5-Coder-0.5B-InstructParams=0.5B2025.12 | 21.1 | 20 | 0 | 18.2 | 17.6 | 0 | 9.1 | 3.2 | 3.6 | 9.5 | 7.8 | 7.1 | 8 | 0 | 39.4 | 0 | 22.4 | 16.8 | 5.7 | 0 | 16.5 | 13.5 | |
| Qwen3-4BParams=4B2025.12 | 21.1 | 16.7 | 0 | 38.1 | 17.6 | 20 | 25.5 | 22.2 | 19.1 | 50 | 28 | 35.7 | 0 | 12.5 | 42.9 | 10.5 | 35.6 | 30.1 | 13 | 50 | 28.7 | 26 | |
| Qwen3-1.7BParams=7B2025.12 | 21.1 | 50 | 0 | 18.2 | 23.5 | 20 | 29.2 | 21.9 | 25.9 | 25 | 27.5 | 21.4 | 0 | 12.5 | 33.3 | 0 | 19.8 | 23.3 | 11.7 | 100 | 19.8 | 20.8 | |
| Qwen3-0.6BParams=0.6B2025.12 | 20 | 16.7 | 0 | 0 | 17.6 | 0 | 12.8 | 6.3 | 10.7 | 8.3 | 8.2 | 7.1 | 0 | 0 | 31 | 15 | 18 | 11.7 | 3.3 | 50 | 13.8 | 11.7 | |
| Qwen2.5-Coder-1.5B-InstructParams=1.5B2025.12 | 20 | 16.7 | 0 | 28.6 | 12.1 | 0 | 30.4 | 19 | 8.8 | 16.7 | 15.4 | 21.4 | 0 | 0 | 44.4 | 10 | 25.7 | 22.1 | 10.8 | 100 | 24.4 | 20.1 | |
| Qwen2.5-1.5B-InstructParams=1.5B2025.12 | 20 | 16.7 | 0 | 18.2 | 23.5 | 0 | 20.8 | 9.4 | 12.1 | 25 | 15.4 | 14.8 | 0 | 25 | 33.3 | 5.3 | 15.8 | 21.4 | 4.3 | 50 | 25.2 | 17.8 | |
| DeepSeek-R1-Distill-Qwen-1.5BParams=1.5B2025.12 | 20 | 16.7 | 0 | 0 | 24.2 | 0 | 12.8 | 9.5 | 3.6 | 8.7 | 3.8 | 0 | 0 | 0 | 11.3 | 0 | 2 | 14.6 | 2.1 | 0 | 5.9 | 7.4 | |
| Qwen2.5-Coder-3B-InstructParams=3B2025.12 | 20 | 54.5 | 0 | 27.3 | 30.3 | 10 | 37.5 | 18.8 | 15.7 | 26.1 | 11.8 | 21.4 | 0 | 25 | 44.4 | 15 | 23.8 | 25.7 | 15 | 50 | 16.9 | 21.3 | |
| Qwen2.5-3B-InstructParams=3B2025.12 | 20 | 18.2 | 0 | 0 | 17.6 | 20 | 29.8 | 15.6 | 13.8 | 16.7 | 19.6 | 14.3 | 0 | 0 | 33.3 | 10.3 | 20 | 22.2 | 9.7 | 100 | 21.5 | 18.6 | |
| DeepSeek-R1-Distill-Qwen-7BParams=7B2025.12 | 20 | 0 | 0 | 18.2 | 18.2 | 0 | 20.8 | 12.5 | 8.7 | 0 | 19.2 | 0 | 0 | 0 | 13.9 | 0 | 14 | 17.6 | 4.3 | 50 | 13.3 | 12.1 | |
| Qwen2.5-7B-InstructParams=7B2025.12 | 20 | 16.7 | 0 | 19 | 35.3 | 20 | 34 | 28.1 | 27.6 | 16.7 | 26.9 | 29.6 | 7.7 | 0 | 36.1 | 10 | 23.8 | 22.7 | 16 | 100 | 22.7 | 23.1 | |
| Hunyuan-7B-InstructParams=7B2025.12 | 20 | 0 | 0 | 9.5 | 17.6 | 20 | 34 | 9.4 | 10.6 | 8.7 | 12.2 | 7.1 | 15.4 | 0 | 29 | 15.4 | 11.9 | 20.5 | 7.5 | 0 | 18.7 | 15.9 | |
| DeepSeek-R1-Distill-Llama-8BParams=8B2025.12 | 20 | 0 | 0 | 9.1 | 11.8 | 10 | 25 | 18.8 | 15.5 | 25 | 19.2 | 14.3 | 0 | 0 | 27.8 | 10 | 17.6 | 19.4 | 6.4 | 50 | 15.1 | 15.6 | |
| DeepSeek-R1-Distill-Qwen-14BParams=14B2025.12 | 20 | 16.7 | 0 | 9.1 | 17.6 | 0 | 37.5 | 25 | 24.1 | 16.7 | 31.4 | 7.1 | 8 | 12.5 | 38.9 | 20 | 23.5 | 25.8 | 13.8 | 50 | 24.9 | 23 | |
| DeepSeek-R1-Distill-Llama-70BParams=70B2025.12 | 20 | 33.3 | 25 | 36.4 | 17.6 | 40 | 41.7 | 31.2 | 37.9 | 25 | 35.3 | 35.7 | 7.7 | 12.5 | 50 | 30 | 35.3 | 34.6 | 23.5 | 50 | 29.6 | 31.9 | |
| DeepSeek-R1-0528-Qwen3-8BParams=8B2025.12 | 11.1 | 18.2 | 0 | 36.4 | 24.2 | 10 | 39.1 | 13.8 | 18.3 | 26.1 | 31.4 | 30.8 | 0 | 0 | 39.4 | 27 | 25 | 30.3 | 12.6 | 66.7 | 27.9 | 24.8 | |
| Llama-3.2-1B-InstructParams=1B2025.12 | 0 | 33.3 | 0 | 0 | 23.5 | 0 | 18.2 | 9.7 | 7 | 19 | 4.1 | 0 | 8 | 25 | 22.9 | 0 | 13.9 | 10.7 | 4.5 | 0 | 14.1 | 10.9 | |
| Llama-3.2-3B-InstructParams=3B2025.12 | 0 | 16.7 | 0 | 20 | 23.5 | 0 | 22.2 | 3.4 | 14.3 | 40 | 8.9 | 14.3 | 0 | 40 | 34.3 | 0 | 16 | 19 | 8.2 | 0 | 16.5 | 15.6 |