Question Answering on BoolQ (test)
91.752AccuracyMistral 7B
Evaluation results
| Method | Links | |
|---|---|---|
| Mistral 7BFramework=Adaptive HPF2024.06 | 91.752 | |
| Gemma 7BFramework=Adaptive HPF2024.06 | 91.166 | |
| Phi-3 3.8BFramework=Adaptive HPF2024.06 | 91.115 | |
| Expanded DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 88.6 | |
| Llama-3 8BFramework=Adaptive HPF2024.06 | 88.577 | |
| BaselineModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 88.5 | |
| Token DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 88.3 | |
| Expert DropModel=Mixtral-8x7B-Instruct, Capacity Factor=1.52025.03 | 87.6 | |
| LLAMA 1Size=65B2023.07 | 85.3 | |
| LLAMA 2Size=70B2023.07 | 85 | |
| LLAMA 2Size=34B2023.07 | 83.7 | |
| Token DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 83.2 | |
| Expanded DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 83.2 | |
| FalconSize=40B2023.07 | 83.1 | |
| LLAMA 1Size=33B2023.07 | 83.1 | |
| BaselineModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 82.9 | |
| LLAMA 2Size=13B2023.07 | 81.7 | |
| Expert DropModel=DeepSeek-V2-Lite-Chat, Capacity Factor=2.02025.03 | 81.6 | |
| BaselineModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 81.3 | |
| Expanded DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 81.1 | |
| Expanded DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 80.9 | |
| Token DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 80.9 | |
| BaselineModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 80.7 | |
| Expert DropModel=Qwen1.5-MoE-Chat, Capacity Factor=1.52025.03 | 80.6 | |
| Token DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 80 | |
| MPTSize=30B2023.07 | 79 | |
| EISAMOptimizer=EISAM(Ours), Architecture=T5-small2026.07 | 78.64 | |
| FSAMOptimizer=FSAM, Architecture=T5-small2026.07 | 78.41 | |
| GSAMOptimizer=GSAM, Architecture=T5-small2026.07 | 78.32 | |
| Expert DropModel=OLMoE-Instruct, Capacity Factor=2.02025.03 | 78.2 | |
| ASAMOptimizer=ASAM, Architecture=T5-small2026.07 | 78.19 | |
| LLAMA 1Size=13B2023.07 | 78.1 | |
| LLAMA 2Size=7B2023.07 | 77.4 | |
| LLAMA 1Size=7B2023.07 | 76.5 | |
| SAMOptimizer=SAM, Architecture=T5-small2026.07 | 76.02 | |
| HyperPELTNumber of Samples=1002022.03 | 75.7 | |
| AdamOptimizer=Adam, Architecture=T5-small2026.07 | 75.18 | |
| Prompt-tuningNumber of Samples=322022.03 | 75.16 | |
| MPTSize=7B2023.07 | 75 | |
| HYPERFORMER++_BASEnumber of samples=20002021.06 | 74.94 | |
| Adapters_BASEnumber of samples=20002021.06 | 73.6 | |
| HYPERFORMER++_BASEnumber of samples=10002021.06 | 72.35 | |
| T5_BASEnumber of samples=20002021.06 | 71.58 | |
| Adapters_BASEnumber of samples=10002021.06 | 70.21 | |
| HYPERFORMER++_BASEnumber of samples=5002021.06 | 70.04 | |
| T5_BASEnumber of samples=10002021.06 | 69.01 | |
| T5-BaseNumber of Samples=10002022.03 | 69.01 | |
| FalconSize=7B2023.07 | 67.5 | |
| Adapters_BASEnumber of samples=5002021.06 | 66.72 | |
| T5_BASEnumber of samples=5002021.06 | 66.49 | |
| HYPERFORMER++_BASEnumber of samples=1002021.06 | 62.03 | |
| T5_BASEnumber of samples=1002021.06 | 60.1 | |
| Adapters_BASEnumber of samples=1002021.06 | 58.6 | |
| T5_BASEnumber of samples=322021.06 | 58.43 | |
| HYPERFORMER++_BASEnumber of samples=322021.06 | 58.37 | |
| T5_BASEnumber of samples=162021.06 | 56.5 | |
| Adapters_BASEnumber of samples=322021.06 | 54.52 | |
| PGM 8 / 8distillation=Before2025.05 | 53.49 | |
| Adapters_BASEnumber of samples=42021.06 | 53.48 | |
| PGM 8 / 8distillation=After Distillation (SDTT)2025.05 | 51.5 | |
| Adapters_BASEnumber of samples=162021.06 | 51.37 | |
| T5_BASEnumber of samples=42021.06 | 50.49 | |
| HYPERFORMER++_BASEnumber of samples=162021.06 | 50.21 | |
| PGM 6 / 6 (1024)distillation=Before2025.05 | 49.82 | |
| MDLMdistillation=Before2025.05 | 49.42 | |
| PGM 6 / 6 (1024)distillation=After Distillation (SDTT)2025.05 | 49.36 | |
| MDLMdistillation=After Distillation (SDTT)2025.05 | 48.59 | |
| HYPERFORMER++_BASEnumber of samples=42021.06 | 48.03 |