Language Modeling on Pile (BPB)
0.74BPBDeepSeekMoE 16B
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeekMoE 16B# Shot=N/A, # Total Params=16.4B, # Activated Params=2.8B, FLOPs per 4K Tokens=74.4T, # Training Tokens=2T2024.01 | 0.74 | |
| DeepSeekMoE 16B# Shot=N/A, # Total Params=16.4B, # Activated Params=2.8B, FLOPs per 4K Tokens=74.4T, # Training Tokens=2T2024.01 | 0.74 | |
| DeepSeek 7B (Dense)# Shot=N/A, # Total Params=6.9B, # Activated Params=6.9B, FLOPs per 4K Tokens=183.5T, # Training Tokens=2T2024.01 | 0.75 | |
| LLaMA2 7B# Shot=N/A, # Total Params=6.7B, # Activated Params=6.7B, FLOPs per 4K Tokens=187.9T, # Training Tokens=2T2024.01 | 0.76 |