Question Answering on GPQA Diamond (Accuracy)
84AccuracyGemini 2.5 Pro
Evaluation Results
| Method | Links | |
|---|---|---|
| Gemini 2.5 ProParameters=Proprietary, Conditions=pass@1 [37]2026.04 | 84 | |
| BF16 (Teacher)sampling=8 samples per problem2026.06 | 73 | |
| QAD (KL-only)sampling=8 samples per problem2026.06 | 72.7 | |
| CKA-QADsampling=8 samples per problem2026.06 | 72.7 | |
| NVFP4 PTQsampling=8 samples per problem2026.06 | 71.6 | |
| PhD domain expertsParameters=—, Conditions=[36]2026.04 | 69.7 | |
| Gemini 2.0 ProParameters=Proprietary, Conditions=pass@1 [35]2026.04 | 64.7 | |
| SST (staged compute imax=4)Parameters=27B, Conditions=0-shot, greedy2026.04 | 61.11 | |
| Gemini 2.0 FlashParameters=Proprietary, Conditions=pass@1 [34]2026.04 | 60.1 | |
| DeepSeek V3Parameters=671B, Conditions=pass@1 [33]2026.04 | 59.1 | |
| GPT-4oParameters=Proprietary, Conditions=pass@1 [32]2026.04 | 56.1 | |
| DCLSize=70B, Selection=Best-of-1002026.04 | 56.06 | |
| GMMSize=70B, Selection=Best-of-1002026.04 | 54.13 | |
| FOREVERModel Scale=14B2026.05 | 54 | |
| DSRL (Ours)Backbone=Qwen3-8B2026.04 | 53.48 | |
| DSSize=70B, Selection=Best-of-1002026.04 | 52.4 | |
| GRPOBackbone=Qwen3-8B2026.04 | 50.96 | |
| Llama 3.3 70B InstructParameters=70B, Conditions=0-shot CoT [31]2026.04 | 50.5 | |
| Qwen 2.5 72B InstructParameters=72B, Conditions=Self-reported [28]2026.04 | 49 | |
| VanillaBackbone=Qwen3-8B2026.04 | 45.86 | |
| Seq. SFTModel Scale=14B2026.05 | 43.4 | |
| EWCModel Scale=14B2026.05 | 43.4 | |
| DSRL (Ours)Backbone=Qwen3-4B2026.04 | 43.18 | |
| Gemma 3 27B-ITParameters=27B, Conditions=5-shot CoT [29]2026.04 | 42.4 | |
| VanillaBackbone=Qwen3-4B2026.04 | 42.12 | |
| GMMSize=8B, Selection=Best-of-1002026.04 | 41.84 | |
| GCWMModel Scale=14B2026.05 | 39.9 | |
| GRPOBackbone=Qwen3-4B2026.04 | 39.39 | |
| AIMMergingModel Scale=14B2026.05 | 38.8 | |
| L&SModel Scale=14B2026.05 | 38.4 | |
| POPBackbone=Qwen-2.5-7B-Inst, Evaluation Protocol=0-shot2026.04 | 38.38 | |
| OPCMModel Scale=14B2026.05 | 38 | |
| BaseBackbone=Qwen-2.5-7B-Inst, Evaluation Protocol=0-shot2026.04 | 36.36 | |
| BaseBackbone=Qwen-2.5-7B, Evaluation Protocol=0-shot2026.04 | 33.84 | |
| Teacher-OPPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 33.6 | |
| MTLModel Scale=14B2026.05 | 33.3 | |
| DAPO + OPPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 33.2 | |
| DSSize=8B, Selection=Best-of-1002026.04 | 32.95 | |
| Train on DBackbone=Qwen-2.5-7B-Inst, Evaluation Protocol=0-shot2026.04 | 32.83 | |
| DCLSize=8B, Selection=Best-of-1002026.04 | 32.79 | |
| Self-OPPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 32.4 | |
| SCALESEARCHATTENTIONModel=Llama 3.1 8B Instruct2026.05 | 32.32 | |
| Instruction-Queryable LoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | 32.3 | |
| Dr.GRPO + OPPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 32 | |
| FULLPRECModel=Llama 3.1 8B Instruct2026.05 | 31.81 | |
| SDPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 30.8 | |
| DAPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 30.4 | |
| Dr.GRPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 29.6 | |
| GRPOBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 29.4 | |
| Queryable LoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | 29.3 | |
| Train on DBackbone=Qwen-2.5-7B, Evaluation Protocol=0-shot2026.04 | 27.78 | |
| POPBackbone=Qwen-2.5-7B, Evaluation Protocol=0-shot2026.04 | 27.78 | |
| DoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | 27.3 | |
| RepLoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | 27.3 | |
| FOREVERModel Scale=1.7B2026.05 | 27.3 | |
| MTLModel Scale=1.7B2026.05 | 26.7 | |
| GCWMModel Scale=1.7B2026.05 | 26.3 | |
| SA3Model=Llama 3.1 8B Instruct2026.05 | 26.26 | |
| Base modelBackbone=Phi-4-mini, Training Framework=DeepScaleR2026.05 | 25.6 | |
| LoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | 25.3 | |
| EWCModel Scale=1.7B2026.05 | 24.8 | |
| OPCMModel Scale=1.7B2026.05 | 23.2 | |
| AIMMergingModel Scale=1.7B2026.05 | 21.7 | |
| L&SModel Scale=1.7B2026.05 | 21.3 | |
| Seq. SFTModel Scale=1.7B2026.05 | 18.2 |