General Language Intelligence on MMLU, GSM8K, BBH, TriviaQA, NQ (test)
83.54MMLUGAC (Qwen2-72B + Llama-3)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| GAC (Qwen2-72B + Llama-3)Date=~2024/06/07, Latency=151.56ms/token2024.06 | 83.54 | 90.91 | 63.99 | 79.29 | 37.65 | 71.08 | 4.06 | |
| GAC (Task-specific Top-2 Ensemble)Date=~2024/06/072024.06 | 83.54 | 90.91 | 63.99 | 80.84 | 37.95 | 71.45 | 4.61 | |
| Qwen2-72B-InstructDate=2024/06/07, Latency=113.91ms/token2024.06 | 82.3 | 89.7 | 62.57 | 73.58 | 33.11 | 68.25 | — | |
| GAC (Llama-3 + Qwen1.5-72B)Date=~2024/04/18, Latency=153.96ms/token2024.06 | 81.49 | 87.06 | 56.73 | 78.6 | 36.01 | 67.98 | -0.47 | |
| GAC (Task-specific Top-2 Ensemble)Date=~2024/04/182024.06 | 81.49 | 87.96 | 58.64 | 80.84 | 37.95 | 69.38 | 1.58 | |
| GAC (Qwen1.5-72B + Yi)Date=~2024/02/04, Latency=103.69ms/token2024.06 | 79.83 | 77.27 | 52.05 | 70.88 | 33.8 | 62.77 | 3.65 | |
| Llama-3-70B-InstructDate=2024/04/18, Latency=150.32ms/token2024.06 | 79.68 | 90 | 57.13 | 79.12 | 35.57 | 68.3 | — | |
| GAC (Qwen1.5-72B + Mixtral)Date=~2024/02/04, Latency=112.83ms/token2024.06 | 79.55 | 75.76 | 54.19 | 75.71 | 31.09 | 63.26 | 4.47 | |
| Qwen1.5-72B-ChatDate=2024/02/04, Latency=102.11ms/token2024.06 | 77.79 | 83.33 | 48.94 | 65.69 | 27.02 | 60.55 | — | |
| GAC (Yi + Mixtral)Date=~2023/12/11, Latency=98.13ms/token2024.06 | 74.83 | 71.21 | 52.64 | 75.6 | 33.52 | 61.56 | 3.13 | |
| GACLatency=98.13ms/token, Ensemble Model Combination=Mixtral and Yi, Ensemble Frequency=every generation step2024.06 | 74.83 | 71.21 | 52.64 | 75.6 | 33.52 | 61.56 | — | |
| OAssistRMLatency=99.75ms/token, Ensemble Model Combination=Mixtral and Yi2024.06 | 73.34 | 70.15 | 51.91 | 72.79 | 30.69 | 59.78 | — | |
| Yi-34B-ChatDate=2023/11/08, Latency=67.96ms/token2024.06 | 72.75 | 68.76 | 50.88 | 70.01 | 29.81 | 58.44 | — | |
| Yi-34B-ChatLatency=67.96ms/token2024.06 | 72.75 | 68.76 | 50.88 | 70.01 | 29.81 | 58.44 | — | |
| LLM Blender (PairRanker)Latency=105.21ms/token, Ensemble Model Combination=Mixtral and Yi2024.06 | 72.69 | 69.59 | 51.7 | 72.37 | 32.24 | 59.72 | — | |
| Mixtral-8x7B-Instruct-v0.1Date=2023/12/11, Latency=96.64ms/token2024.06 | 70.89 | 66.82 | 49.84 | 76.54 | 34.35 | 59.69 | — | |
| Mixtral-8x7B-Instruct-v0.1Latency=96.64ms/token2024.06 | 70.89 | 66.82 | 49.84 | 76.54 | 34.35 | 59.69 | — | |
| UltraRMLatency=114.57ms/token, Ensemble Model Combination=Mixtral and Yi2024.06 | 69.49 | 71.09 | 52.27 | 73.82 | 32.36 | 59.81 | — | |
| GACLatency=51.32ms/token, Ensemble Model Combination=openchat and SOLAR, Ensemble Frequency=every generation step2024.06 | 66.51 | 74.3 | 51.19 | 72.5 | 33.82 | 59.66 | — | |
| GACLatency=31.34ms/token, Ensemble Model Combination=openchat and SOLAR, Ensembled token proportion=7.68%2024.06 | 65.14 | 73.18 | 50.32 | 69.68 | 31.75 | 58.01 | — | |
| UltraRMLatency=74.51ms/token, Ensemble Model Combination=openchat and SOLAR2024.06 | 64.94 | 75.51 | 50.65 | 71.23 | 32.03 | 58.87 | — | |
| Nous-Hermes-2-SOLAR-10.7BLatency=50.89ms/token2024.06 | 64.88 | 72.86 | 49.92 | 71.33 | 32.21 | 58.24 | — | |
| OAssistRMLatency=55.62ms/token, Ensemble Model Combination=openchat and SOLAR2024.06 | 64.87 | 72.93 | 49.02 | 70.06 | 31.64 | 57.7 | — | |
| LLM Blender (PairRanker)Latency=57.86ms/token, Ensemble Model Combination=openchat and SOLAR2024.06 | 64.23 | 74.07 | 50.18 | 70.55 | 32.55 | 58.32 | — | |
| FuseLLMLatency=28.31ms/token, Ensemble Model Combination=openchat and SOLAR2024.06 | 63.94 | 65.5 | 46.32 | 64.57 | 29.06 | 53.88 | — | |
| openchat_3.5Latency=28.01ms/token2024.06 | 63.87 | 68.46 | 47.96 | 68.12 | 29.75 | 55.63 | — |