Machine Translation on IWSLT (test)
5.3PerplexityImitKD
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| ImitKDArchitecture=SRU (3-layer), Training Data=Full2020.09 | 5.3 | — | 0.348 | 0.354 | |
| VanillaArchitecture=SRU (3-layer), Training Data=Full2020.09 | 5.4 | — | 0.342 | 0.348 | |
| TeacherArchitecture=Transformer (8-layer)2020.09 | 5.6 | — | 0.344 | 0.352 | |
| ImitKDArchitecture=SRU (2-layer), Training Data=Full2020.09 | 5.7 | — | 0.337 | 0.345 | |
| VanillaArchitecture=Transformer (2-layer), Training Data=Full2020.09 | 5.9 | — | 0.338 | 0.342 | |
| ImitKDArchitecture=Transformer (2-layer), Training Data=Full2020.09 | 5.9 | — | 0.344 | 0.348 | |
| VanillaArchitecture=SRU (2-layer), Training Data=Full2020.09 | 6 | — | 0.33 | 0.338 | |
| SeqKDArchitecture=SRU (3-layer), Training Data=Full2020.09 | 6.1 | — | 0.343 | 0.348 | |
| VanillaArchitecture=Transformer (2-layer)2020.09 | 6.4 | — | 0.328 | 0.334 | |
| SeqKDArchitecture=SRU (2-layer), Training Data=Full2020.09 | 6.8 | — | 0.331 | 0.337 | |
| SeqKDArchitecture=Transformer (2-layer), Training Data=Full2020.09 | 7.1 | — | 0.34 | 0.344 | |
| VanillaArchitecture=SRU (3-layer)2020.09 | 7.4 | — | 0.3 | 0.312 | |
| VanillaArchitecture=SRU (2-layer)2020.09 | 7.4 | — | 0.295 | 0.306 | |
| ImitKDArchitecture=Transformer (2-layer)2020.09 | 7.5 | — | 0.343 | 0.346 | |
| ImitKDArchitecture=SRU (2-layer)2020.09 | 12.7 | — | 0.333 | 0.337 | |
| ImitKDArchitecture=SRU (3-layer)2020.09 | 14.7 | — | 0.341 | 0.344 | |
| SeqKDArchitecture=Transformer (2-layer)2020.09 | 23.4 | — | 0.34 | 0.342 | |
| SeqKDArchitecture=SRU (2-layer)2020.09 | 102.1 | — | 0.32 | 0.324 | |
| SeqKDArchitecture=SRU (3-layer)2020.09 | 153 | — | 0.33 | 0.331 | |
| Baseline# Softmaxes=12018.09 | — | 0.2741 | — | — | |
| BaselineLR Schedule=Baseline, Training Epochs=502020.03 | — | 0.3497 | — | — | |
| BPE-MoS# Softmaxes=92018.09 | — | 0.2891 | — | — | |
| Gemma 7BFramework=Adaptive HPF, Threshold=0.152024.06 | — | 0.0845 | — | — | |
| Gemma 7BFramework=Adaptive HPF, Threshold=0.22024.06 | — | 0.0773 | — | — | |
| Hybrid-LightRNN-MoS# Softmaxes=92018.09 | — | 0.2879 | — | — | |
| KneeLR Schedule=Knee, Training Epochs=502020.03 | — | 0.3553 | — | — | |
| Knee (short budget)LR Schedule=Knee (short budget), Training Epochs=352020.03 | — | 0.3508 | — | — | |
| Llama-3 8BFramework=Adaptive HPF, Threshold=0.152024.06 | — | 0.2114 | — | — | |
| Llama-3 8BFramework=Adaptive HPF, Threshold=0.22024.06 | — | 0.2115 | — | — | |
| Mistral 7BFramework=Adaptive HPF, Threshold=0.152024.06 | — | 0.2 | — | — | |
| Mistral 7BFramework=Adaptive HPF, Threshold=0.22024.06 | — | 0.2057 | — | — | |
| Phi-3 3.8BFramework=Adaptive HPF, Threshold=0.152024.06 | — | 0.1556 | — | — | |
| Phi-3 3.8BFramework=Adaptive HPF, Threshold=0.22024.06 | — | 0.1535 | — | — |