Zero-shot Evaluation on Downstream NLU tasks 1.0 (eval)
57.90-shot ScoreRandom Sampling KD (Ours 12+)
Evaluation results
| Method | Links | |
|---|---|---|
| Random Sampling KD (Ours 12+)Unique Tokens=12, Adaptive Weighting=true2025.03 | 57.9 | |
| Random Sampling KD (Ours 12)Unique Tokens=122025.03 | 57.5 | |
| FullKD2025.03 | 57.5 | |
| Top-K 50K=502025.03 | 57.1 | |
| Top-K 12K=122025.03 | 56.6 | |
| CE2025.03 | 55.6 |