Few-shot Language Understanding Suite (HS, WG, OBQA, ARC-E, ARC-C, PIQA, SIQA, Story)
0.555HS AccuracyTAD
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| TAD#Tkn=+2B, K=5, Teacher=Phi3 4B2026.02 | 0.555 | 0.602 | 0.352 | 0.636 | 0.326 | 0.714 | 0.449 | 0.694 | 0.541 | |
| TinyLlama(TL)-1.1B#Tkn=2T2026.02 | 0.552 | 0.589 | 0.334 | 0.613 | 0.307 | 0.714 | 0.421 | 0.689 | 0.527 | |
| TAD#Tkn=+2B, K=1, Teacher=Phi3 4B2026.02 | 0.551 | 0.6 | 0.348 | 0.642 | 0.331 | 0.713 | 0.448 | 0.691 | 0.54 | |
| TAD#Tkn=+2B, K=20, Teacher=Phi3 4B2026.02 | 0.548 | 0.599 | 0.338 | 0.625 | 0.32 | 0.721 | 0.442 | 0.687 | 0.535 | |
| TAD#Tkn=+2B, K=10, Teacher=Phi3 4B2026.02 | 0.546 | 0.6 | 0.346 | 0.632 | 0.324 | 0.718 | 0.446 | 0.689 | 0.538 | |
| Vanilla KD#Tkn=+2B, Teacher=Phi3 4B2026.02 | 0.541 | 0.589 | 0.335 | 0.632 | 0.316 | 0.712 | 0.442 | 0.687 | 0.532 | |
| CLM (no KD)#Tkn=+2B2026.02 | 0.538 | 0.571 | 0.326 | 0.618 | 0.304 | 0.711 | 0.414 | 0.685 | 0.521 | |
| TinyLlama(TL)-1.1B#Tkn=1T2026.02 | 0.536 | 0.568 | 0.322 | 0.612 | 0.301 | 0.708 | 0.412 | 0.681 | 0.518 |