Language Model Evaluation on Downstream Tasks Evaluation Suite (MMLU Components)
4.92Math AccuracyMath Dense Expert
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Math Dense ExpertCategory=Specialized Models2024.08 | 4.92 | 12.39 | 68.21 | 13.32 | 46.11 | 34.29 | 29.87 | |
| BAM DM (Expert KV)Category=Generalist Models, KV Strategy=Expert KV2024.08 | 4.44 | 12.83 | 85.47 | 19.89 | 47.11 | 34.42 | 34.02 | |
| BAM CM (Expert KV)Category=Generalist Models, KV Strategy=Expert KV2024.08 | 4.34 | 12.48 | 82.79 | 19.51 | 47.43 | 34.43 | 33.5 | |
| BAM DM (Shared KV)Category=Generalist Models, KV Strategy=Shared KV2024.08 | 4.1 | 11.76 | 86.73 | 19.48 | 47.27 | 34.55 | 33.98 | |
| BTX (Baseline)Category=Generalist Models2024.08 | 3.86 | 10.05 | 81.85 | 19.07 | 47.36 | 34.07 | 32.71 | |
| Seed ModelParameters=2B2024.08 | 3.68 | 9.41 | 73.34 | 21.33 | 47.73 | 34.13 | 31.6 | |
| BAM CM (Shared KV)Category=Generalist Models, KV Strategy=Shared KV2024.08 | 3.65 | 11.77 | 80.98 | 19.22 | 47.56 | 34.16 | 32.89 | |
| Code Dense ExpertCategory=Specialized Models2024.08 | 3.19 | 18.8 | 21.49 | 12.18 | 44.29 | 31.5 | 21.91 | |
| Law Dense ExpertCategory=Specialized Models2024.08 | 3.05 | 0.2 | 88.8 | 10.41 | 44.08 | 32.18 | 29.79 |