Language Modeling on SNLG and SNLU evaluation suites (test)
78.71SNLG ScoreTeacher baseline (OPT-6.7B)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Teacher baseline (OPT-6.7B)Model Role=Teacher, Backbone=OPT-6.7B2024.02 | 78.71 | 23.43 | 51.07 | |
| Teacher baseline (OPT-2.7B)Model Role=Teacher, Backbone=OPT-2.7B2024.02 | 74.21 | 22.28 | 48.25 | |
| Teacher baseline (OPT-1.3B)Model Role=Teacher, Backbone=OPT-1.3B2024.02 | 68.9 | 22.6 | 45.75 | |
| ImitKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 62.66 | 19.56 | 41.11 | |
| GKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 62.62 | 19.26 | 40.94 | |
| GKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 62.46 | 19.21 | 40.84 | |
| GKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 61.36 | 19.07 | 40.21 | |
| f-distill + ATKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 61.3 | 19.54 | 40.42 | |
| f-distill + ATKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 61.25 | 19.18 | 40.22 | |
| GKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 61.24 | 17.48 | 39.36 | |
| GKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 61.23 | 18.77 | 40 | |
| ImitKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 60.77 | 19.18 | 39.97 | |
| ImitKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 60.76 | 19.29 | 40.02 | |
| f-distill + ATKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 60.7 | 19.02 | 39.86 | |
| GKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 60.59 | 16.87 | 38.73 | |
| f-distillStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 60.01 | 17.08 | 38.55 | |
| ImitKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 59.88 | 17.46 | 38.67 | |
| ImitKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 59.87 | 18.41 | 39.14 | |
| f-distillStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 59.74 | 19.46 | 39.6 | |
| f-distillStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 59.02 | 17.8 | 38.41 | |
| ImitKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 58.86 | 17.28 | 38.07 | |
| Teacher baseline (OPT-350M)Model Role=Teacher, Backbone=OPT-350M2024.02 | 58.33 | 20.36 | 39.35 | |
| Supervised KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 57.56 | 19.31 | 38.43 | |
| Supervised KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 57.26 | 19.48 | 38.37 | |
| Supervised KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 56.76 | 19.73 | 38.25 | |
| Supervised KDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 55.57 | 17.99 | 36.78 | |
| Supervised KDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 55.45 | 18.33 | 36.89 | |
| Supervised KDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 55.3 | 18.69 | 37 | |
| Reverse KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 54.4 | 19.4 | 36.9 | |
| Reverse KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 54.37 | 19.16 | 36.76 | |
| Reverse KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 54.34 | 19.27 | 36.8 | |
| f-distill + ATKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 52.69 | 18.8 | 35.75 | |
| ImitKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 52.36 | 18.66 | 35.51 | |
| ImitKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 52.27 | 18.35 | 35.31 | |
| f-distillStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 52.18 | 18.57 | 35.37 | |
| Supervised KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 52.16 | 19.58 | 35.87 | |
| GKD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 51.9 | 18.52 | 35.21 | |
| GKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 51.87 | 17.32 | 34.59 | |
| Reverse KDStudent Model=OPT-125M, Teacher Model=OPT-1.3B2024.02 | 51.6 | 18.15 | 34.87 | |
| Reverse KDStudent Model=OPT-125M, Teacher Model=OPT-2.7B2024.02 | 51.26 | 18.56 | 34.91 | |
| Reverse KD + ATKDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 50.86 | 19.13 | 34.99 | |
| Supervised KDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 50.62 | 18.88 | 34.75 | |
| Reverse KDStudent Model=OPT-125M, Teacher Model=OPT-350M2024.02 | 50.54 | 18.05 | 34.3 | |
| Reverse KDStudent Model=OPT-125M, Teacher Model=OPT-6.7B2024.02 | 50.08 | 18.33 | 34.2 |