Reasoning on BBH, MMLU, ARC-C, and ThmQA (test)
64.66BBHTeacher
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| TeacherModel=Qwen2.5-7B2026.05 | 64.66 | 78.22 | 89.9 | 32.47 | 66.31 | |
| TeacherTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 57.72 | 70.9 | 83.58 | 18.1 | 57.58 | |
| TeacherTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 52.57 | 66.17 | 81.23 | 26.12 | 56.52 | |
| HybKDTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 46.53 | 69.05 | 81.23 | 23.82 | 55.16 | |
| HybKDTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 46.53 | 69.05 | 81.23 | 23.82 | 55.16 | |
| Hybrid KDSetting=Qwen2.5-32B→3B2026.05 | 45.7 | 66.8 | 80 | 24 | 54.1 | |
| JS divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 45.5 | 64.68 | 78.85 | 22.27 | 52.83 | |
| JS divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 45.5 | 64.68 | 78.85 | 22.27 | 52.83 | |
| α-β divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 45.12 | 64.95 | 79.81 | 22.94 | 53.21 | |
| α-β divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 45.12 | 64.95 | 79.81 | 22.94 | 53.21 | |
| Adaptive KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 44.71 | 64.69 | 79.23 | 22.25 | 52.72 | |
| Adaptive KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 44.71 | 64.69 | 79.23 | 22.25 | 52.72 | |
| Reverse KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 44.07 | 65.67 | 77.68 | 24.2 | 52.9 | |
| Reverse KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 44.07 | 65.67 | 77.68 | 24.2 | 52.9 | |
| Soft KDSetting=Qwen2.5-32B→3B2026.05 | 44 | 65.6 | 78.1 | 22.9 | 52.6 | |
| Forward KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 41.65 | 64.45 | 78.33 | 23.02 | 51.87 | |
| Hard KDTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 41.52 | 65.76 | 78.75 | 23.75 | 52.45 | |
| Skew FKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 41.39 | 64.67 | 77.75 | 23.77 | 51.89 | |
| Skew FKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 41.39 | 64.67 | 77.75 | 23.77 | 51.89 | |
| Skew RKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 41.22 | 63.95 | 76.91 | 23.67 | 51.44 | |
| Skew RKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 41.22 | 63.95 | 76.91 | 23.67 | 51.44 | |
| Total VariationTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 40.5 | 64.52 | 78.11 | 22.83 | 51.49 | |
| Total VariationTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 40.5 | 64.52 | 78.11 | 22.83 | 51.49 | |
| Hard KDSetting=Qwen2.5-32B→3B2026.05 | 34.4 | 67.2 | 79.5 | 23.1 | 51 | |
| Skew RKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 28.16 | 45.03 | 47.51 | 11.37 | 33.02 | |
| Skew RKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 28.16 | 45.03 | 47.51 | 11.37 | 33.02 | |
| HybKDTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 27.44 | 35.64 | 37.01 | 5 | 26.27 | |
| Total VariationTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.74 | 44.35 | 46.76 | 10.2 | 32.01 | |
| Total VariationTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.74 | 44.35 | 46.76 | 10.2 | 32.01 | |
| HybKDTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.58 | 49.08 | 51.69 | 10.5 | 34.46 | |
| HybKDTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.58 | 49.08 | 51.69 | 10.5 | 34.46 | |
| α-β divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.18 | 42.59 | 46.7 | 11.07 | 31.63 | |
| α-β divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.18 | 42.59 | 46.7 | 11.07 | 31.63 | |
| Adaptive KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.04 | 41.85 | 45.89 | 11.33 | 31.28 | |
| Adaptive KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 26.04 | 41.85 | 45.89 | 11.33 | 31.28 | |
| Skew FKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 25.87 | 44.12 | 47.12 | 10.65 | 31.94 | |
| Skew FKLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 25.87 | 44.12 | 47.12 | 10.65 | 31.94 | |
| Skew RKLTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 25.24 | 33 | 32.24 | 4.27 | 23.69 | |
| Adaptive KLTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 25.18 | 33.86 | 33.74 | 4.85 | 24.41 | |
| JS divergenceTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 25.16 | 34.08 | 34.42 | 5.7 | 24.84 | |
| α-β divergenceTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 25.07 | 34.78 | 34.9 | 5.27 | 25.01 | |
| Skew FKLTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 25.05 | 31.39 | 32.34 | 4.73 | 23.37 | |
| Reverse KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 24.91 | 44.72 | 47.44 | 11 | 32.02 | |
| Reverse KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 24.91 | 44.72 | 47.44 | 11 | 32.02 | |
| JS divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 24.55 | 43.31 | 44.78 | 11.82 | 31.12 | |
| JS divergenceTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 24.55 | 43.31 | 44.78 | 11.82 | 31.12 | |
| Forward KLTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 24.42 | 43.19 | 46.84 | 10.33 | 31.2 | |
| Reverse KLTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 23.69 | 31.63 | 32.08 | 3.6 | 22.75 | |
| Total VariationTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 23.55 | 27.41 | 28.99 | 2.68 | 20.66 | |
| Student (No Distill)Teacher Model=Qwen2.5-7B, Student Model=Qwen2.5-3B2026.05 | 22.34 | 64.61 | 78.4 | 12.22 | 44.39 | |
| Forward KLTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 22.07 | 33.13 | 33.41 | 4.37 | 23.25 | |
| Hard KDTeacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 15.29 | 22.54 | 23.98 | 3.88 | 16.42 | |
| Student (No Distill)Teacher-Student Model Pair=Llama3.1-8B → Llama3.2-1B2026.05 | 14.01 | 19.78 | 21.57 | 2.22 | 14.4 | |
| Hard KDTeacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 10.27 | 36.83 | 40.58 | 11.35 | 24.76 | |
| Reverse KLTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 9.52 | 24.77 | 27.58 | 1.26 | 15.78 | |
| HybKDTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 7.78 | 25.94 | 26.07 | 5.24 | 16.26 | |
| α-β divergenceTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 7.26 | 25.49 | 26.07 | 4.34 | 15.79 | |
| JS divergenceTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 7.14 | 25.27 | 26.03 | 4.57 | 15.75 | |
| Skew FKLTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 7.01 | 25.08 | 26.14 | 5.01 | 15.81 | |
| Student (No Distill)Teacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 6.47 | 15.94 | 15.7 | 0.95 | 9.76 | |
| Total VariationTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 6.29 | 25.07 | 25.83 | 1.29 | 14.62 | |
| Forward KLTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 5.91 | 24.05 | 25 | 4.25 | 14.8 | |
| Skew RKLTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 5.83 | 25.17 | 25.93 | 3.38 | 15.08 | |
| Adaptive KLTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 5.47 | 24.76 | 24.07 | 4.23 | 14.63 | |
| Hard KDTeacher-Student Model Pair=Gemma3-4B → Gemma3-1B2026.05 | 5.28 | 18.11 | 18.09 | 3.15 | 11.16 | |
| Student (No Distill)Teacher Model=Qwen2.5-7B, Student Model=Qwen2.5-0.5B2026.05 | 3.81 | 44.67 | 43.72 | 7.78 | 24.99 |