Exact Match (EM) on TriviaQA (val)
16.6Exact MatchDeepSeekMoE
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepSeekMoE# Shot=5-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.01 | 16.6 | |
| GShard# Shot=5-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.01 | 10.2 | |
| Switch# Shot=5-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 8.9 | |
| Hash Layer# Shot=5-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 6.5 | |
| Dense# Shot=5-shot, # Total Params=0.2B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 4.9 |