Downstream Task on 11 Downstream Tasks Aggregate
64.6Average AccuracyLLaMA2-7B
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA2-7B# tokens for training=2T, Training data source=Different from RedPajama2023.10 | 64.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Sheared-LLaMA-2.7B# tokens for training=50B, Training data source=RedPajama2023.10 | 56.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Sheared-LLaMA-1.3B# tokens for training=50B, Training data source=RedPajama2023.10 | 51 | — | — | — | — | — | — | — | — | — | — | — | |
| TinyLlama-1.1BFFN Size=100%2025.06 | 47.18 | — | — | — | — | — | — | — | — | — | — | — | |
| DIVE 1/8retraining budget=15B tokens, ARC-c shots=25, HellaSwag shots=10, MMLU shots=5, active experts=1/82025.06 | 42.52 | 83 | 68.01 | 54.7 | 52.53 | 24.4 | 22.55 | 37.17 | 19.35 | 59.88 | 20.2 | 25.88 | |
| DIVE 1/8FFN Size=50% × 12025.06 | 42.17 | — | — | — | — | — | — | — | — | — | — | — | |
| FLAPFFN Size=50%2025.06 | 41.42 | — | — | — | — | — | — | — | — | — | — | — | |
| DIVE 2/8retraining budget=15B tokens, ARC-c shots=25, HellaSwag shots=10, MMLU shots=5, active experts=2/82025.06 | 41.12 | 81.5 | 64.96 | 54.78 | 47.1 | 23.12 | 22.58 | 33.27 | 21.97 | 58.47 | 18.2 | 26.33 | |
| DIVE 2/8FFN Size=25% × 22025.06 | 40.92 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-MoE 2/8retraining budget=15B tokens, ARC-c shots=25, HellaSwag shots=10, MMLU shots=5, active experts=2/82025.06 | 39.77 | 77.9 | 62.84 | 53.51 | 42.89 | 22.01 | 23.08 | 32.23 | 21.51 | 61.77 | 14.6 | 25.14 | |
| LLaMA-MoE 1/8retraining budget=15B tokens, ARC-c shots=25, HellaSwag shots=10, MMLU shots=5, active experts=1/82025.06 | 39.65 | 77.4 | 61.15 | 52.81 | 43.56 | 22.1 | 21.07 | 33.73 | 20.58 | 61.71 | 16.6 | 25.47 | |
| LLM-PrunerFFN Size=50%2025.06 | 39.57 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-MoE 1/8FFN Size=50% × 12025.06 | 39.34 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA-MoE 2/8FFN Size=25% × 22025.06 | 39.24 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=760M, Evaluation Protocol=Zero-shot2025.02 | 39.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=760M, Evaluation Protocol=Zero-shot2025.02 | 38.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=760M, Evaluation Protocol=Zero-shot2025.02 | 37.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=760M, Evaluation Protocol=Zero-shot2025.02 | 37.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=355M, Evaluation Protocol=Zero-shot2025.02 | 37.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=355M, Evaluation Protocol=Zero-shot2025.02 | 36.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=355M, Evaluation Protocol=Zero-shot2025.02 | 36.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=760M, Evaluation Protocol=Zero-shot2025.02 | 36 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=355M, Evaluation Protocol=Zero-shot2025.02 | 35.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=760M, Evaluation Protocol=Zero-shot2025.02 | 35.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=355M, Evaluation Protocol=Zero-shot2025.02 | 35.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=355M, Evaluation Protocol=Zero-shot2025.02 | 35 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=125M, Evaluation Protocol=Zero-shot2025.02 | 35 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=125M, Evaluation Protocol=Zero-shot2025.02 | 34.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=125M, Evaluation Protocol=Zero-shot2025.02 | 34.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=125M, Evaluation Protocol=Zero-shot2025.02 | 34.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=125M, Evaluation Protocol=Zero-shot2025.02 | 34 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=125M, Evaluation Protocol=Zero-shot2025.02 | 33.9 | — | — | — | — | — | — | — | — | — | — | — |