Language Modeling on Domain-Specific Corpora Pretrain, Code, Law, Math
32.69Pretrain ScoreLaw Dense Expert
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Law Dense ExpertParameters=2B, Expert Type=Law2024.08 | 32.69 | 6.84 | 3.09 | 8.61 | 6.18 | |
| Math Dense ExpertParameters=2B, Expert Type=Math2024.08 | 20.32 | 3.2 | 5.11 | 3.2 | 3.84 | |
| Code Dense ExpertParameters=2B, Expert Type=Code2024.08 | 15.39 | 2.18 | 5.22 | 4.34 | 3.91 | |
| BTX (Baseline)Parameters=2B2024.08 | 10.35 | 2.4 | 3.76 | 3.64 | 3.27 | |
| BAM CMParameters=2B, Mechanism=CM, KV Strategy=Shared KV2024.08 | 10.28 | 2.38 | 3.72 | 3.61 | 3.24 | |
| BAM DMParameters=2B, Mechanism=DM, KV Strategy=Shared KV2024.08 | 10.2 | 2.37 | 3.69 | 3.59 | 3.22 | |
| BAM CMParameters=2B, Mechanism=CM, KV Strategy=Expert KV2024.08 | 10.19 | 2.37 | 3.69 | 3.57 | 3.21 | |
| BAM DMParameters=2B, Mechanism=DM, KV Strategy=Expert KV2024.08 | 10.11 | 2.36 | 3.66 | 3.55 | 3.19 | |
| Base ModelParameters=2B2024.08 | 9.83 | 2.41 | 3.86 | 3.71 | 3.33 |