Natural Language Generation on E2E NLG Challenge
70.6BLEUDual LoRA (r = 4)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Dual LoRA (r = 4)Model=GPT2_M, Trainable Parameters (M)=0.702025.12 | 70.6 | 8.86 | 46.9 | 72.4 | 2.56 | |
| Dual LoRA (r = 4)Model=GPT2_L, Trainable Parameters (M)=1.542025.12 | 70.6 | 8.86 | 47.1 | 72.5 | 2.54 | |
| GPT-2 M (LoRA)Model=GPT-2 Medium, Adaptation Method=LoRA, # Trainable Parameters=0.35M2021.06 | 70.4 | 8.85 | 46.8 | 71.8 | 2.53 | |
| GPT-2 L (LoRA)Model=GPT-2 Large, Adaptation Method=LoRA, # Trainable Parameters=0.77M2021.06 | 70.4 | 8.89 | 46.8 | 72 | 2.47 | |
| GPT-2 L (PreLayer)Model=GPT-2 Large, Adaptation Method=Prefix-Tuning, # Trainable Parameters=0.77M2021.06 | 70.3 | 8.85 | 46.2 | 71.7 | 2.47 | |
| PrefixModel=GPT2_L, Trainable Parameters (M)=0.772025.12 | 70.3 | 8.85 | 46.2 | 71.7 | 2.47 | |
| LoRA (r = 4)Model=GPT2_L, Trainable Parameters (M)=0.772025.12 | 70.3 | 8.85 | 46.8 | 71.9 | 2.52 | |
| LoRA+ (r = 8)Model=GPT2_M, Trainable Parameters (M)=0.702025.12 | 70.2 | 8.81 | 46.6 | 71.6 | 2.53 | |
| LoRA (r = 8)Model=GPT2_L, Trainable Parameters (M)=1.542025.12 | 70 | 8.8 | 46.8 | 71.7 | 2.54 | |
| LoRA+ (r = 8)Model=GPT2_L, Trainable Parameters (M)=1.542025.12 | 70 | 8.83 | 46.8 | 71.9 | 2.53 | |
| LoRA (r = 8)Model=GPT2_M, Trainable Parameters (M)=0.702025.12 | 69.9 | 8.77 | 46.8 | 71.7 | 2.5 | |
| DoRA (r = 8)Model=GPT2_L, Trainable Parameters (M)=1.562025.12 | 69.8 | 8.78 | 46.6 | 71.6 | 2.52 | |
| HiddenKeyModel=GPT2-Medium2024.02 | 69.76 | 8.765 | 46.8 | 71.78 | 2.511 | |
| GPT-2 M (PreLayer)Model=GPT-2 Medium, Adaptation Method=Prefix-Tuning, # Trainable Parameters=0.35M2021.06 | 69.7 | 8.81 | 46.1 | 71.4 | 2.49 | |
| PrefixModel=GPT2_M, Trainable Parameters (M)=0.352025.12 | 69.7 | 8.81 | 46.1 | 71.4 | 2.49 | |
| DoRA (r = 8)Model=GPT2_M, Trainable Parameters (M)=0.712025.12 | 69.5 | 8.75 | 46.4 | 71.4 | 2.52 | |
| HiddenKey¯Model=GPT2-Medium2024.02 | 69.35 | 8.726 | 46.6 | 71.61 | 2.51 | |
| HiddenCutModel=GPT2-Medium2024.02 | 69.22 | 8.7 | 46.66 | 71.39 | 2.491 | |
| GPT-2 L (Adapter^H)Model=GPT-2 Large, Adaptation Method=Adapter (Houlsby), # Trainable Parameters=0.88M2021.06 | 69.1 | 8.68 | 46.3 | 71.4 | 2.49 | |
| Adapter-LModel=GPT2_L, Trainable Parameters (M)=23.002025.12 | 69.1 | 8.68 | 46.3 | 71.4 | 2.49 | |
| HiddenKeyModel=LLAMA2-7B2024.02 | 69.02 | 8.725 | 45.84 | 71.17 | 2.456 | |
| GPT-2 M (Adapter^L) [Prior]Model=GPT-2 Medium, Adaptation Method=Adapter (Lin), # Trainable Parameters=11.09M2021.06 | 68.9 | 8.71 | 46.1 | 71.3 | 2.47 | |
| GPT-2 L (Adapter^L)Model=GPT-2 Large, Adaptation Method=Adapter (Lin), # Trainable Parameters=23.00M2021.06 | 68.9 | 8.7 | 46.1 | 71.3 | 2.45 | |
| Adapter-LModel=GPT2_M, Trainable Parameters (M)=11.092025.12 | 68.9 | 8.71 | 46.1 | 71.3 | 2.47 | |
| LoRA (r = 4)Model=GPT2_M, Trainable Parameters (M)=0.352025.12 | 68.9 | 8.69 | 46.4 | 71.4 | 2.52 | |
| DropKeyModel=GPT2-Medium2024.02 | 68.78 | 8.651 | 46.53 | 71.4 | 2.486 | |
| GPT-2 L (FT)Model=GPT-2 Large, Adaptation Method=Full Fine-Tuning, # Trainable Parameters=774.03M2021.06 | 68.5 | 8.78 | 46 | 69.9 | 2.45 | |
| BaselineModel=GPT2-Medium2024.02 | 68.5 | 8.615 | 46.43 | 71.08 | 2.49 | |
| FFTModel=GPT2_L, Trainable Parameters (M)=774.032025.12 | 68.5 | 8.78 | 46 | 69.9 | 2.45 | |
| LoRABackbone=GPT-2 large, # Param=1.5M2024.02 | 68.24 | 8.76 | 46.23 | 69.92 | 2.42 | |
| GPT-2 M (FT)Model=GPT-2 Medium, Adaptation Method=Full Fine-Tuning, # Trainable Parameters=354.92M2021.06 | 68.2 | 8.62 | 46.2 | 71 | 2.47 | |
| Full Finetuning*Model=GPT2-Medium2024.02 | 68.2 | 8.62 | 46.2 | 71 | 2.47 | |
| FFTModel=GPT2_M, Trainable Parameters (M)=354.922025.12 | 68.2 | 8.62 | 46.2 | 71 | 2.47 | |
| GPT-2 M (FT_Top2)Model=GPT-2 Medium, Adaptation Method=Fine-Tuning Top 2 Layers, # Trainable Parameters=25.19M2021.06 | 68.1 | 8.59 | 46 | 70.8 | 2.41 | |
| 4-bit AdamWBackbone=GPT-2 M, Precision=4-bit2023.09 | 67.8 | — | — | — | — | |
| 32-bit AdamWBackbone=GPT-2 M, Precision=32-bit2023.09 | 67.7 | — | — | — | — | |
| 4-bit FactorBackbone=GPT-2 M, Precision=4-bit2023.09 | 67.6 | — | — | — | — | |
| 8-bit AdamW†Backbone=GPT-2 M, Precision=8-bit, Quantization Policy=Do not quantize optimizer states for embedding layers2023.09 | 67.5 | — | — | — | — | |
| LoRABackbone=GPT-2 medium, # Param=0.8M2024.02 | 67.43 | 8.65 | 46.01 | 69.64 | 2.42 | |
| GPT-2 M (Adapter^L) [Re-run]Model=GPT-2 Medium, Adaptation Method=Adapter (Lin), # Trainable Parameters=11.09M2021.06 | 67.3 | 8.5 | 46 | 70.7 | 2.44 | |
| 32-bit AdafactorBackbone=GPT-2 M, Precision=32-bit2023.09 | 67.2 | — | — | — | — | |
| 32-bit Adafactor+Backbone=GPT-2 M, Precision=32-bit, beta1=02023.09 | 67.2 | — | — | — | — | |
| 32-bit SM3Backbone=GPT-2 M, Precision=32-bit2023.09 | 66.9 | — | — | — | — | |
| BaselineModel=LLAMA2-7B2024.02 | 66.71 | 8.463 | 44.82 | 70.1 | 2.371 | |
| GPT-2 M (Adapter^H)Model=GPT-2 Medium, Adaptation Method=Adapter (Houlsby), # Trainable Parameters=0.37M2021.06 | 66.3 | 8.41 | 45 | 69.8 | 2.4 | |
| FTBackbone=GPT-2 medium, # Param=355M2024.02 | 65.95 | 8.52 | 45.95 | 69.13 | 2.35 | |
| FT top2Backbone=GPT-2 medium, # Param=25.2M2024.02 | 65.94 | 8.53 | 44.28 | 68.83 | 2.23 | |
| AdapterBackbone=GPT-2 large, # Param=1.8M2024.02 | 65.94 | 8.46 | 45.78 | 68.65 | 2.34 | |
| REDBackbone=GPT-2 large, # Param=0.09M2024.02 | 65.77 | 8.42 | 46.12 | 69.03 | 2.36 | |
| FTBackbone=GPT-2 large, # Param=774M2024.02 | 65.56 | 8.5 | 45.4 | 68.38 | 2.27 | |
| Adapter-FFNBackbone=GPT-2 large, # Param=1.5M2024.02 | 65.53 | 8.41 | 45.65 | 68.46 | 2.33 | |
| Prefix TuningBackbone=GPT-2 large, # Param=1.5M2024.02 | 65.5 | 8.45 | 43.97 | 67.32 | 2.23 | |
| IA3Backbone=GPT-2 large, # Param=0.32M2024.02 | 65.08 | 8.5 | 42.72 | 66.8 | 2.15 | |
| REDBackbone=GPT-2 medium, # Param=0.05M2024.02 | 64.86 | 8.36 | 44.99 | 67.62 | 2.28 | |
| Adapter-FFNBackbone=GPT-2 medium, # Param=0.8M2024.02 | 64.41 | 8.3 | 44.74 | 67.53 | 2.29 | |
| AdapterBackbone=GPT-2 medium, # Param=0.9M2024.02 | 64.31 | 8.29 | 44.91 | 67.72 | 2.28 | |
| Prefix TuningBackbone=GPT-2 medium, # Param=0.8M2024.02 | 63.92 | 8.26 | 41.81 | 66.86 | 2.03 | |
| IA3Backbone=GPT-2 medium, # Param=0.17M2024.02 | 63.63 | 7.99 | 40.49 | 66.36 | 1.89 |