Natural Language Understanding on CME benchmark
27.6OAI SumHydraLoRA
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HydraLoRAMoE=true, Rank=32, Time=34h, Params%=1.842024.08 | 27.6 | 95.9 | 57.8 | 86.5 | 87.2 | 70.1 | 70.2 | 50.6 | 24.6 | 37 | 42.2 | 59.06 | |
| TeamLoRAMoE=true, Rank=32, Time=29h, Params%=2.712024.08 | 27.6 | 95.7 | 58.9 | 87.5 | 87.1 | 73.8 | 72.3 | 51.8 | 26.4 | 38.8 | 43.3 | 60.29 | |
| AdaLoRAMoE=false, Rank=128, Time=30h, Params%=2.562024.08 | 27.4 | 95.5 | 57.2 | 87 | 86.3 | 72.1 | 71.1 | 46.8 | 25.5 | 35.2 | 42.9 | 58.82 | |
| MoELoRAMoE=true, Rank=32, Time=42h, Params%=2.712024.08 | 27.4 | 95.5 | 59.3 | 87.2 | 86.1 | 72.9 | 71.8 | 50.1 | 25.1 | 38.4 | 42.8 | 59.69 | |
| TeamLoRAMoE=true, Rank=16, Time=28h, Params%=1.352024.08 | 27.4 | 95.9 | 59.2 | 86.6 | 87 | 73.1 | 73.1 | 51.3 | 25.9 | 37.1 | 42.8 | 59.95 | |
| LoRAMoE=false, Rank=128, Time=26h, Params%=2.682024.08 | 27.3 | 95.6 | 56.8 | 87.4 | 85.7 | 71.6 | 70.8 | 47.2 | 25.2 | 36.8 | 42.5 | 58.81 | |
| MoSLoRAMoE=false, Rank=128, Time=28h, Params%=2.72024.08 | 27.3 | 95.6 | 58.3 | 86.8 | 86.6 | 73.2 | 71.9 | 47.4 | 25.8 | 38.4 | 41.4 | 59.34 | |
| LoRAMoE=false, Rank=32, Time=25h, Params%=0.672024.08 | 27.2 | 95.6 | 57.6 | 84.9 | 87 | 65.8 | 68.2 | 47.1 | 23.3 | 34.7 | 40.4 | 57.44 | |
| IA3MoE=false, Time=24h, Params%=0.032024.08 | 26.4 | 92 | 48.7 | 78.3 | 68.1 | 61.5 | 55.1 | 37.7 | 18.8 | 19.5 | 34.9 | 49.18 | |
| Prompt-TuningMoE=false, Time=23h, Params%=0.022024.08 | 25.3 | 91.1 | 44.2 | 77 | 65.4 | 59.7 | 54.8 | 38.7 | 16.2 | 19.4 | 31.2 | 47.55 | |
| TeamLoRABase Model=Llama-3-8B, Rank (r)=82024.08 | 25.2 | 94.2 | 49.7 | 79 | 81.4 | 58.3 | 70.1 | 36.1 | 22.2 | 41.3 | 52.1 | 55.42 | |
| MoELoRABase Model=Llama-3-8B, Rank (r)=82024.08 | 24.8 | 94.9 | 47.6 | 79 | 81 | 58.2 | 69.8 | 35.1 | 20.2 | 40.4 | 49.2 | 54.56 | |
| LoRABase Model=Llama-3-8B, Rank (r)=322024.08 | 24.3 | 95.1 | 47.2 | 78.9 | 80.1 | 58.3 | 70.6 | 34.6 | 19.3 | 37.1 | 52.2 | 54.34 |