Pairwise Solution Preference Prediction on Data-centric Solution Preference Dataset (test)
61.5Average AccuracyDeepSeek-V3.2 (Thinking mode)
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V3.2 (Thinking mode)Thinking mode=true, Temperature=1, Reasoning instructions=Wei et al., 2023; Kojima et al., 20232026.01 | 61.5 | 59.3 | 66.9 | 57.4 | 60.4 | 63.9 | 57 | 58.9 | 62.1 | 66.8 | |
| GPT-5.1Version=gpt-5.1-2025-11-13, Temperature=1, Reasoning instructions=Wei et al., 2023; Kojima et al., 20232026.01 | 58.8 | 55.4 | 63 | 57.4 | 55.5 | 61.6 | 59.7 | 57.2 | 59.2 | 62.2 | |
| Complexity Heuristicdimensions=Code Engineering, Model Architecture, Data Pipeline2026.01 | 50.8 | — | — | — | — | — | — | — | — | — | |
| Random Guess2026.01 | 50 | — | — | — | — | — | — | — | — | — |