Language Understanding on MMLU (Subject Averages)
68.6Humanities AvgMixtral-8x22B
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Mixtral-8x22BNumber of experts=82024.07 | 68.6 | 67.1 | 84.1 | 78.7 | — | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B, Prompting=zero-shot CoT2025.06 | 64.59 | 81.01 | 82.22 | 77.61 | 74.99 | |
| Model-centricNumber of experts=62024.07 | 63.7 | 62.1 | 80 | 75.6 | — | |
| Data-centricNumber of experts=62024.07 | 62.3 | 60.2 | 78.5 | 73.4 | — | |
| TableDreamer-27KBackbone=Llama3.1-8B, Prompting=zero-shot CoT2025.06 | 61.42 | 71.24 | 75.43 | 74.95 | 69.73 | |
| Llama3.1-8B-InstructBackbone=Llama3.1-8B, Prompting=zero-shot CoT2025.06 | 61.11 | 69.52 | 75.76 | 75.35 | 69.41 | |
| Mixtral-8x7BNumber of experts=8, Base Model=Mixtral-8x7B2024.07 | 60.5 | 58.9 | 77.8 | 74.2 | — | |
| EnumerateNumber of experts=62024.07 | 60.4 | 59.5 | 78 | 73 | — | |
| TableGPT2Backbone=Qwen2.5-7B, Prompting=zero-shot CoT2025.06 | 59.49 | 75.08 | 78.55 | 74.43 | 70.47 | |
| Model-centricNumber of experts=42024.07 | 58.1 | 54.3 | 72.5 | 68.3 | — | |
| Count-guidedNumber of experts=62024.07 | 58 | 54.1 | 74.9 | 70.2 | — | |
| Data-centricNumber of experts=42024.07 | 57.8 | 51.3 | 69.7 | 64.2 | — | |
| Data-centricNumber of experts=6, Base Model=Mixtral-8x7B2024.07 | 56 | 52.4 | 73.1 | 68.2 | — | |
| Model-centricNumber of experts=6, Base Model=Mixtral-8x7B2024.07 | 54.4 | 51.8 | 70.2 | 66.8 | — | |
| EnumerateNumber of experts=42024.07 | 53.9 | 52.3 | 67.2 | 64.2 | — | |
| EnumerateNumber of experts=6, Base Model=Mixtral-8x7B2024.07 | 52.4 | 49 | 66.4 | 63.7 | — | |
| Router-guidedNumber of experts=6, Base Model=Mixtral-8x7B2024.07 | 51.8 | 46.9 | 60.5 | 60.5 | — | |
| Full-FTBase Model=LLaMa2-7B, Evaluation Protocol=Fine-tuning, Number of random seeds=32024.05 | 49.9 | 41.7 | 57.5 | 57 | 51.5 | |
| OWSBase Model=LLaMa2-7B, Evaluation Protocol=Fine-tuning, Number of random seeds=32024.05 | 49.8 | 42.1 | 58.6 | 59.7 | 52.6 | |
| Count-guidedNumber of experts=6, Base Model=Mixtral-8x7B2024.07 | 49.2 | 45 | 59.7 | 58.2 | — | |
| Model-centricNumber of experts=4, Base Model=Mixtral-8x7B2024.07 | 48.1 | 45.2 | 58.5 | 55.2 | — | |
| Data-centricNumber of experts=4, Base Model=Mixtral-8x7B2024.07 | 48 | 43.3 | 57 | 54.6 | — | |
| LoRABase Model=LLaMa2-7B, Evaluation Protocol=Fine-tuning, Number of random seeds=32024.05 | 46.1 | 40.8 | 56.6 | 56.2 | 49.9 | |
| Count-guidedNumber of experts=42024.07 | 45.7 | 42 | 57.7 | 45.7 | — | |
| GaLoreBase Model=LLaMa2-7B, Evaluation Protocol=Fine-tuning, Number of random seeds=32024.05 | 45.4 | 41.7 | 55.8 | 56 | 49.7 | |
| LISABase Model=LLaMa2-7B, Evaluation Protocol=Fine-tuning, Number of random seeds=32024.05 | 44.9 | 41.2 | 54.7 | 57.6 | 49.6 | |
| EnumerateNumber of experts=4, Base Model=Mixtral-8x7B2024.07 | 43.5 | 40.4 | 52.7 | 43.5 | — | |
| Count-guidedNumber of experts=4, Base Model=Mixtral-8x7B2024.07 | 36.9 | 35.1 | 45.6 | 43.4 | — | |
| Router-guidedNumber of experts=62024.07 | 27.3 | 24.4 | 25.4 | 27.9 | — | |
| Router-guidedNumber of experts=4, Base Model=Mixtral-8x7B2024.07 | 24.8 | 24.7 | 26.5 | 25 | — | |
| Router-guidedNumber of experts=42024.07 | 22.7 | 24 | 25.8 | 23.4 | — | |
| TableBenchLLMBackbone=Llama3.1-8B, Prompting=zero-shot CoT2025.06 | 15.75 | 12.62 | 22.59 | 11.29 | 15.55 | |
| TableLLMBackbone=CodeLlama-7B, Prompting=zero-shot CoT2025.06 | 12.55 | 16.29 | 16.83 | 16.09 | 17.3 |