Mathematical Reasoning on GSM8K, MATH, SVAMP, ASDiv, MAWPS, TAB, MQA, SAT (test)
56GSM8K AccuracyTAD
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| TADK=1, Teacher=Phi3-14B, Student=Llama2-7B, Data=OWM(2.5B) + ORCAMEL2026.02 | 56 | 10.2 | 77.2 | 77.1 | 91.8 | 39.8 | 39.2 | 40.6 | 54 | |
| TADK=20, Teacher=Phi3-14B, Student=Llama2-7B, Data=OWM(2.5B) + ORCAMEL2026.02 | 52.8 | 8 | 77.6 | 76.9 | 92.4 | 39.2 | 39 | 46.9 | 54.1 | |
| TADK=5, Teacher=Phi3-14B, Student=Llama2-7B, Data=OWM(2.5B) + ORCAMEL2026.02 | 51.6 | 9.2 | 76.7 | 75.4 | 91.2 | 38.7 | 40.5 | 37.5 | 52.6 | |
| TADK=10, Teacher=Phi3-14B, Student=Llama2-7B, Data=OWM(2.5B) + ORCAMEL2026.02 | 51.4 | 8.4 | 76.6 | 75.5 | 90.6 | 38.7 | 39.2 | 44.4 | 53.1 | |
| Vanilla KDTeacher=Phi3-14B, Student=Llama2-7B, Data=OWM(2.5B) + ORCAMEL2026.02 | 50.5 | 8.1 | 75.3 | 74.4 | 90.5 | 29.7 | 37.2 | 34.4 | 50 | |
| WizardMath-7BData=+RL with Evol Instruct2026.02 | 46.6 | 7 | 56.8 | 65.2 | 81.1 | 35 | 20.3 | 23.1 | 41.9 | |
| Orca2-7BData=+SFT (ORCA) + KTO2026.02 | 40 | 6.2 | 70.2 | 67 | 87.5 | 30.4 | 31.6 | 28.1 | 45.1 | |
| Llemma-7BData=+ProofPile(0.2T)2026.02 | 39.7 | 15.4 | 56.9 | 67.7 | 83.3 | 47 | 40.9 | 44 | 49.4 | |
| TADK=1, Teacher=Phi3-4B, Student=TinyLlama-1.1B, Data=OWM(2.5B) + ORCAMEL2026.02 | 36.8 | 6.8 | 67.8 | 67.9 | 81.7 | 25.4 | 16.3 | 28.1 | 41.4 | |
| Rho-1-Math(1.1B)Data=+OWM (30B)2026.02 | 36.3 | 13.4 | 52.6 | 66.5 | 83.6 | 29.5 | 32.1 | 18.5 | 41.5 | |
| TADK=5, Teacher=Phi3-4B, Student=TinyLlama-1.1B, Data=OWM(2.5B) + ORCAMEL2026.02 | 33.2 | 7.4 | 65.4 | 68.7 | 85.6 | 27.6 | 17.9 | 34.4 | 42.5 | |
| Vanilla KDTeacher=Phi3-4B, Student=TinyLlama-1.1B, Data=OWM(2.5B) + ORCAMEL2026.02 | 30.8 | 6.8 | 64.6 | 62.5 | 80.7 | 20.1 | 16.7 | 27.5 | 38.7 | |
| TADK=10, Teacher=Phi3-4B, Student=TinyLlama-1.1B, Data=OWM(2.5B) + ORCAMEL2026.02 | 30.1 | 9 | 65.7 | 68.4 | 85.4 | 24.1 | 18.2 | 29.8 | 41.3 | |
| TADK=20, Teacher=Phi3-4B, Student=TinyLlama-1.1B, Data=OWM(2.5B) + ORCAMEL2026.02 | 28.2 | 7.2 | 66.2 | 68.2 | 84.2 | 24.6 | 17.1 | 25 | 40.1 | |
| Llama2-7BData=CLM + SFT + OWM(2.5B) + ORCAMEL2026.02 | 22 | 4.2 | 47.7 | 56.3 | 72.3 | 37.7 | 23 | 28.1 | 36.4 | |
| TinyLlama(TL)-1.1BData=CLM + SFT + OWM(2.5B) + ORCAMEL2026.02 | 19.6 | 4 | 49.4 | 58.8 | 74.3 | 21.8 | 18 | 28.1 | 34.3 | |
| Llama2-7BData=Web (2T)2026.02 | 14.2 | 3.6 | 39.1 | 51.6 | 63.6 | 30.9 | 12.5 | 32.8 | 31.4 | |
| TADTop-K (K)=5, Data (#Tkns)=+ OWM(2.5B), Teacher=Phi3 4B, Student=TinyLlama-1.1B, Distillation Strategy=Tail-Aware Distillation, Evaluation Protocol=few-shot CoT2026.02 | 7.1 | 4.8 | 19.2 | 35.9 | 46.7 | 15.9 | 10 | 22.6 | 20.3 | |
| TADTop-K (K)=20, Data (#Tkns)=+ OWM(2.5B), Teacher=Phi3 4B, Student=TinyLlama-1.1B, Distillation Strategy=Tail-Aware Distillation, Evaluation Protocol=few-shot CoT2026.02 | 6.5 | 3.8 | 18.2 | 31.7 | 40.9 | 13.7 | 9 | 31.2 | 19.4 | |
| Llama3.2-1.2B-PTData (#Tkns)=Web (9T), Evaluation Protocol=few-shot CoT2026.02 | 6.5 | 4.2 | 21.7 | 35.7 | 44.2 | 21.1 | 13.2 | 6.2 | 19.1 | |
| TADTop-K (K)=10, Data (#Tkns)=+ OWM(2.5B), Teacher=Phi3 4B, Student=TinyLlama-1.1B, Distillation Strategy=Tail-Aware Distillation, Evaluation Protocol=few-shot CoT2026.02 | 6.4 | 4.6 | 19.7 | 33 | 42.7 | 12.9 | 9.3 | 37.5 | 20.7 | |
| Vanilla KDData (#Tkns)=+ OWM(2.5B), Teacher=Phi3 4B, Student=TinyLlama-1.1B, Distillation Strategy=Vanilla KD, Evaluation Protocol=few-shot CoT2026.02 | 6.1 | 4.2 | 21.1 | 33.5 | 41.5 | 15.5 | 11.2 | 16.7 | 18.7 | |
| TADTop-K (K)=1, Data (#Tkns)=+ OWM(2.5B), Teacher=Phi3 4B, Student=TinyLlama-1.1B, Distillation Strategy=Tail-Aware Distillation, Evaluation Protocol=few-shot CoT2026.02 | 6.1 | 6.2 | 22.1 | 33.1 | 41.5 | 14 | 11.3 | 21.9 | 19.5 | |
| CLM (no KD)Data (#Tkns)=+ OWM(2.5B), Student=TinyLlama-1.1B, Evaluation Protocol=few-shot CoT2026.02 | 3.9 | 3.8 | 17.9 | 29.7 | 39.5 | 12.2 | 10.8 | 15.6 | 16.7 | |
| MiniPLMData (#Tkns)=+ OWM(2.5B), Evaluation Protocol=few-shot CoT2026.02 | 3.3 | 3.4 | 13.4 | 27.3 | 34 | 10.8 | 10.5 | 12.5 | 14.4 | |
| Gemma3-1B-PTData (#Tkns)=Web (2T), Evaluation Protocol=few-shot CoT2026.02 | 2.1 | 2.2 | 12.8 | 17.1 | 22.4 | 11.1 | 14.5 | 15.6 | 12.2 | |
| TinyLlama(TL)-1.1BData=Web (2.5T)2026.02 | 2 | 2.6 | 9.5 | 16.3 | 20.1 | 12.7 | 12.8 | 15.6 | 11.4 | |
| TinyLlama(TL)-1.1BData (#Tkns)=Web (2.5T), Evaluation Protocol=few-shot CoT2026.02 | 2 | 2.6 | 9.5 | 16.3 | 20.1 | 12.7 | 12.8 | 15.6 | 11.4 |