Reasoning on GPQA Diamond (Accuracy, Efficiency, Modification)
61.6Accuracy32B model
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| 32B modelSpeculative Model=32B, Protocol=pass@12025.04 | 61.6 | — | 5,406.8 | — | 31.6 | — | — | |
| 14B modelSpeculative Model=14B, Protocol=pass@12025.04 | 57.1 | — | 5,762.7 | — | 57.8 | — | — | |
| Speculative ThinkingSpeculative Model=7B, Target Model=32B, Protocol=pass@12025.04 | 52 | 6.5 | 5,952.5 | -2.6 | 40.3 | 27.5 | 22 | |
| 7B modelSpeculative Model=7B, Protocol=pass@12025.04 | 45.5 | — | 6,111.5 | — | 62.1 | — | — | |
| Speculative ThinkingSpeculative Model=1.5B, Target Model=32B, Protocol=pass@12025.04 | 41.9 | 8.1 | 7,612.4 | -3.9 | 91.8 | 190.4 | 17 | |
| Speculative ThinkingSpeculative Model=1.5B, Target Model=14B, Protocol=pass@12025.04 | 38.9 | 5.1 | 8,134.3 | 2.7 | 128.1 | 121.7 | 15 | |
| Deepseek-Distilled Qwen-2.5-1.5BSpeculative Model=1.5B, Protocol=pass@12025.04 | 33.8 | — | 7,922 | — | 223.2 | — | — |