Language Modeling on SlimPajama
2.97Perplexity (PPL)D³
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| D³Granularity=Ours, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 2.97 | — | — | — | — | — | — | — | — | |
| Dynamic LossGranularity=Sample-level, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 3.1 | — | — | — | — | — | — | — | — | |
| Uniform SamplingGranularity=Sample-level, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 3.13 | — | — | — | — | — | — | — | — | |
| DoReMiGranularity=Domain-level, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 3.3 | — | — | — | — | — | — | — | — | |
| DoGEGranularity=Domain-level, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 3.31 | — | — | — | — | — | — | — | — | |
| Data Mixing LawGranularity=Domain-level, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 4.5 | — | — | — | — | — | — | — | — | |
| RegMixGranularity=Domain-level, Architecture=GPT-2 Medium (355M), Training Tokens=100B2026.05 | 4.51 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-2-13B (MHA-RoPE), Compression Ratio=10%2025.05 | 6.11 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-3.1-70B (GQA-RoPE), Compression Ratio=10%2025.05 | 6.59 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-2-7B (MHA-RoPE), Compression Ratio=10%2025.05 | 6.68 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-2-13B (MHA-RoPE), Compression Ratio=20%2025.05 | 7.15 | — | — | — | — | — | — | — | — | |
| A3Compression=10%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 7.28 | — | — | — | — | — | — | — | 0.58 | |
| DRPruningFrom=7B, To=2.7B2024.11 | 7.4 | — | — | — | — | — | — | — | — | |
| Sheared LlamaFrom=7B, To=2.7B2024.11 | 7.64 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-30B, CRatio=20%2025.05 | 7.71 | — | — | — | — | — | — | — | — | |
| ReShearedFrom=7B, To=2.7B2024.11 | 7.83 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-2-7B (MHA-RoPE), Compression Ratio=20%2025.05 | 7.91 | — | — | — | — | — | — | — | — | |
| PaluModel=MPT-30B, CRatio=20%2025.05 | 7.91 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=LLaMA-2-13B (MHA-RoPE), Compression Ratio=10%2025.05 | 7.95 | — | — | — | — | — | — | — | — | |
| A3Compression=20%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 8.16 | — | — | — | — | — | — | — | 1.46 | |
| A3Model=MPT-30B, CRatio=40%2025.05 | 8.33 | — | — | — | — | — | — | — | — | |
| SVD-LLMCompression=10%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 8.4 | — | — | — | — | — | — | — | 1.69 | |
| SVD-LLMModel=LLaMA-3.1-70B (GQA-RoPE), Compression Ratio=10%2025.05 | 8.43 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-7B, CRatio=20%2025.05 | 8.88 | — | — | — | — | — | — | — | — | |
| PaluModel=MPT-30B, CRatio=40%2025.05 | 8.99 | — | — | — | — | — | — | — | — | |
| MoEUTnumber of parameters=1040M, zero-shot=true2024.05 | 9.36 | 38 | 82.5 | 90.2 | 38.1 | 64.6 | 39.1 | 58.7 | — | |
| SVD-LLMModel=LLaMA-2-7B (MHA-RoPE), Compression Ratio=10%2025.05 | 9.49 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-3.1-8B (GQA-RoPE), Compression Ratio=10%2025.05 | 9.52 | — | — | — | — | — | — | — | — | |
| Baselinenumber of parameters=1040M, zero-shot=true2024.05 | 9.56 | 38.8 | 80.5 | 89.9 | 37.6 | 64.5 | 38.7 | 58.3 | — | |
| A³Model=Ministral-3-8B, Compression=10%2025.05 | 9.66 | — | — | — | — | — | — | — | — | |
| SVD-LLMCompression=20%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 9.67 | — | — | — | — | — | — | — | 2.96 | |
| PaluModel=MPT-7B, CRatio=20%2025.05 | 9.67 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=LLaMA-2-13B (MHA-RoPE), Compression Ratio=20%2025.05 | 9.75 | — | — | — | — | — | — | — | — | |
| DRPruningFrom=7B, To=1.3B2024.11 | 9.83 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-7B, CRatio=40%2025.05 | 9.9 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-3.1-70B (GQA-RoPE), Compression Ratio=20%2025.05 | 10.02 | — | — | — | — | — | — | — | — | |
| Sheared LlamaFrom=7B, To=1.3B2024.11 | 10.05 | — | — | — | — | — | — | — | — | |
| ReShearedFrom=7B, To=1.3B2024.11 | 10.42 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=LLaMA-3.1-70B (GQA-RoPE), Compression Ratio=20%2025.05 | 10.44 | — | — | — | — | — | — | — | — | |
| MoEUTnumber of parameters=244M, zero-shot=true2024.05 | 11.47 | 30.7 | 80.2 | 86.8 | 32 | 61.7 | 35.8 | 54.5 | — | |
| Baselinenumber of parameters=244M, zero-shot=true2024.05 | 11.51 | 31.9 | 78.6 | 87.3 | 31.7 | 60.9 | 36.6 | 54.5 | — | |
| PaluModel=MPT-7B, CRatio=40%2025.05 | 11.51 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-30B, CRatio=20%2025.05 | 11.52 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-30B, CRatio=60%2025.05 | 11.52 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=LLaMA-2-7B (MHA-RoPE), Compression Ratio=20%2025.05 | 11.93 | — | — | — | — | — | — | — | — | |
| A³Model=Ministral-3-3B, Compression=10%2025.05 | 12.15 | — | — | — | — | — | — | — | — | |
| A³Model=Ministral-3-8B, Compression=20%2025.05 | 12.66 | — | — | — | — | — | — | — | — | |
| OPT-1.3BTraining tokens=300B2026.03 | 13.11 | — | — | — | — | — | — | — | — | |
| OPT-1.3B FFFTraining tokens=26B, FFF depth (d)=4, Training regime=Fine-tuning (FT)2026.03 | 13.37 | — | — | — | — | — | — | — | — | |
| A³Model=LLaMA-3.1-8B (GQA-RoPE), Compression Ratio=20%2025.05 | 13.58 | — | — | — | — | — | — | — | — | |
| OPT-1.3B FFFTraining tokens=26B, FFF depth (d)=6, Training regime=Fine-tuning (FT)2026.03 | 14.11 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=Ministral-3-8B, Compression=10%2025.05 | 14.72 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=LLaMA-3.1-8B (GQA-RoPE), Compression Ratio=10%2025.05 | 15.14 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-7B, CRatio=60%2025.05 | 15.34 | — | — | — | — | — | — | — | — | |
| PaluModel=MPT-30B, CRatio=60%2025.05 | 15.59 | — | — | — | — | — | — | — | — | |
| MoEUTnumber of parameters=44M, zero-shot=true2024.05 | 15.77 | 19.8 | 75.9 | 82.1 | 28 | 57.5 | 32.1 | 49.2 | — | |
| OPT-1.3BTraining tokens=26B2026.03 | 15.83 | — | — | — | — | — | — | — | — | |
| Baselinenumber of parameters=44M, zero-shot=true2024.05 | 16.42 | 20 | 72.8 | 80.7 | 27.5 | 57 | 31.6 | 48.3 | — | |
| OPT-1.3B FFFTraining tokens=26B, FFF depth (d)=4, Training regime=From scratch2026.03 | 16.61 | — | — | — | — | — | — | — | — | |
| OPT-1.3B FFFTraining tokens=26B, FFF depth (d)=6, Training regime=From scratch2026.03 | 17.57 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-30B, CRatio=40%2025.05 | 18 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=Ministral-3-3B, Compression=10%2025.05 | 18.27 | — | — | — | — | — | — | — | — | |
| A³Model=Ministral-3-3B, Compression=20%2025.05 | 18.53 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=Ministral-3-8B, Compression=20%2025.05 | 20.28 | — | — | — | — | — | — | — | — | |
| PaluModel=MPT-7B, CRatio=60%2025.05 | 25.73 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=Ministral-3-3B, Compression=20%2025.05 | 26.49 | — | — | — | — | — | — | — | — | |
| SVD-LLMModel=LLaMA-3.1-8B (GQA-RoPE), Compression Ratio=20%2025.05 | 27.44 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-30B, CRatio=80%2025.05 | 37.09 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-7B, CRatio=20%2025.05 | 48.11 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-30B, CRatio=60%2025.05 | 54.97 | — | — | — | — | — | — | — | — | |
| PaluModel=MPT-30B, CRatio=80%2025.05 | 211 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-7B, CRatio=40%2025.05 | 383 | — | — | — | — | — | — | — | — | |
| A3Model=MPT-7B, CRatio=80%2025.05 | 388 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-30B, CRatio=80%2025.05 | 779 | — | — | — | — | — | — | — | — | |
| PaluModel=MPT-7B, CRatio=80%2025.05 | 5,270 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-7B, CRatio=60%2025.05 | 5,397 | — | — | — | — | — | — | — | — | |
| CloverModel=MPT-7B, CRatio=80%2025.05 | 15,467 | — | — | — | — | — | — | — | — |