Language Modeling on Domain-Specific Corpora Pretrain, Code, Law, Math (test)
87.82Pretrain ScoreLaw Dense Expert
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Law Dense ExpertModel Architecture=Specialized, Number of Parameters=590M, Domain=Law2024.08 | 87.82 | 23.45 | 7.53 | 30.79 | 37.4 | |
| Math Dense ExpertModel Architecture=Specialized, Number of Parameters=590M, Domain=Math2024.08 | 57.61 | 6.78 | 10.43 | 5.67 | 20.12 | |
| Code Dense ExpertModel Architecture=Specialized, Number of Parameters=590M, Domain=Code2024.08 | 39.5 | 3.71 | 10.83 | 8.77 | 15.7 | |
| BTX (Baseline)Model Architecture=Generalist, Number of Parameters=590M2024.08 | 26.72 | 3.78 | 6.63 | 5.77 | 10.72 | |
| BAM CM (Shared KV)Model Architecture=Generalist, Number of Parameters=590M, KV Cache Strategy=Shared KV, Mixing Strategy=Coupled (CM)2024.08 | 26 | 3.72 | 6.5 | 5.63 | 10.46 | |
| BAM DM (Shared KV)Model Architecture=Generalist, Number of Parameters=590M, KV Cache Strategy=Shared KV, Mixing Strategy=Decoupled (DM)2024.08 | 25.67 | 3.64 | 6.39 | 5.57 | 10.32 | |
| BAM CM (Expert KV)Model Architecture=Generalist, Number of Parameters=590M, KV Cache Strategy=Expert KV, Mixing Strategy=Coupled (CM)2024.08 | 24.84 | 3.53 | 6.21 | 5.43 | 10 | |
| BAM DM (Expert KV)Model Architecture=Generalist, Number of Parameters=590M, KV Cache Strategy=Expert KV, Mixing Strategy=Decoupled (DM)2024.08 | 24.02 | 3.45 | 6.04 | 5.31 | 9.7 | |
| Base ModelModel Architecture=Base, Number of Parameters=590M2024.08 | 23.98 | 8.62 | 8.92 | 14.12 | 13.91 |