Language Modeling on SlimPajama (test)
0.5402PPL (CommonCrawl)MIXChunk
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| MIXChunkSequence Length (L)=8K2024.02 | 0.5402 | 0.4867 | 0.6219 | 0.7443 | 0.582 | 0.5042 | 0.6531 | 0.5903 | — | |
| MIXChunkSequence Length (L)=2K2024.02 | 0.5429 | 0.495 | 0.6238 | 0.7665 | 0.5974 | 0.5001 | 0.6406 | 0.5952 | — | |
| UNIChunkSequence Length (L)=8K2024.02 | 0.5429 | 0.4888 | 0.6235 | 0.7483 | 0.5859 | 0.5065 | 0.6564 | 0.5932 | — | |
| UNIChunkSequence Length (L)=2K2024.02 | 0.5468 | 0.4984 | 0.6298 | 0.7709 | 0.6011 | 0.5033 | 0.6436 | 0.5991 | — | |
| BM25ChunkSequence Length (L)=8K2024.02 | 0.5489 | 0.4952 | 0.6391 | 0.7621 | 0.5919 | 0.5108 | 0.6599 | 0.6011 | — | |
| BM25ChunkSequence Length (L)=2K2024.02 | 0.5496 | 0.5021 | 0.6394 | 0.7782 | 0.6041 | 0.505 | 0.6452 | 0.6034 | — | |
| INTRADocSequence Length (L)=8K2024.02 | 0.5506 | 0.4988 | 0.6443 | 0.7643 | 0.5936 | 0.5119 | 0.6597 | 0.6033 | — | |
| INTRADocSequence Length (L)=2K2024.02 | 0.5507 | 0.5048 | 0.6426 | 0.7793 | 0.605 | 0.5062 | 0.6458 | 0.6049 | — | |
| INTRADocMaximum sequence length (L)=8K, Backbone=LLaMA-1.3B2024.02 | 8.994 | 13.173 | 6.073 | 5.01 | 7.894 | 10.701 | 4.705 | 8.079 | — | |
| BM25ChunkMaximum sequence length (L)=8K, Backbone=LLaMA-1.3B2024.02 | 9.144 | 13.579 | 6.287 | 5.463 | 8.022 | 10.81 | 4.715 | 8.289 | — | |
| UNIChunkMaximum sequence length (L)=8K, Backbone=LLaMA-1.3B2024.02 | 9.478 | 14.19 | 6.897 | 7.006 | 8.456 | 11.117 | 4.812 | 8.851 | — | |
| MIXChunkMaximum sequence length (L)=8K, Backbone=LLaMA-1.3B2024.02 | 9.645 | 14.424 | 7.01 | 7.496 | 8.634 | 11.337 | 4.911 | 9.065 | — | |
| BM25ChunkMaximum sequence length (L)=2K, Backbone=LLaMA-1.3B2024.02 | 11.418 | 13.677 | 6.237 | 4.585 | 7.623 | 11.253 | 5.059 | 8.55 | — | |
| INTRADocMaximum sequence length (L)=2K, Backbone=LLaMA-1.3B2024.02 | 11.631 | 13.197 | 6.084 | 4.252 | 7.535 | 11.13 | 5.03 | 8.41 | — | |
| UNIChunkMaximum sequence length (L)=2K, Backbone=LLaMA-1.3B2024.02 | 11.805 | 13.65 | 6.546 | 5.518 | 7.839 | 11.353 | 5.106 | 8.831 | — | |
| MIXChunkMaximum sequence length (L)=2K, Backbone=LLaMA-1.3B2024.02 | 13.284 | 13.884 | 6.811 | 5.531 | 8.051 | 11.623 | 5.203 | 9.172 | — | |
| Sheared LlamaModel Size=1.3B2024.11 | 21.75 | 26 | 29.5 | 93.75 | 56.5 | 33.5 | 38.5 | 42.79 | — | |
| ReShearedModel Size=1.3B2024.11 | 30.5 | 30.25 | 23 | 89.25 | 47.5 | 32 | 81 | 47.64 | — | |
| ConstantModel Size=1.3B2024.11 | 35 | 40 | 21.75 | 88 | 72.5 | 47.75 | 82 | 55.29 | — | |
| TANDEMModel=Pythia 3B, Data Regime=data restricted2026.06 | 39.59 | 43.6 | 29.2 | 7.49 | 17.26 | 42.85 | 15.73 | 23.81 | — | |
| DRPruningModel Size=1.3B2024.11 | 44 | 51.5 | 33.5 | 94.75 | 90 | 48 | 86.5 | 64.04 | — | |
| UniformModel=Pythia 3B, Data Regime=data restricted2026.06 | 55.45 | 60.14 | 41.76 | 9.85 | 22.08 | 53.83 | 17.2 | 31.03 | — | |
| ReShearedModel Size=2.7B2024.11 | 61.75 | 60.25 | 80.5 | 96 | 92 | 73 | 93.75 | 79.61 | — | |
| Sheared LlamaModel Size=2.7B2024.11 | 81.25 | 89.5 | 89.25 | 95.5 | 82.75 | 96.5 | 90.5 | 89.32 | — | |
| DRPruningModel Size=2.7B2024.11 | 82.25 | 77.75 | 87.5 | 99 | 89.25 | 86.75 | 79.5 | 86 | — | |
| TransformerTotal parameters=918m, Non-embedding parameters=764m2026.01 | — | — | — | — | — | — | — | — | 2.912 | |
| Transformer + Spelling Bee EmbeddingsTotal parameters=918m, Non-embedding parameters=764m2026.01 | — | — | — | — | — | — | — | — | 2.897 |