Training Efficiency on Large Language Model Pre-training
46.2Model FLOPs UtilizationPaLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PaLM# of Parameters (in billions)=540B, Accelerator chips=6144 TPU v42022.04 | 46.2 | 57.8 | |
| Gopher# of Parameters (in billions)=280B, Accelerator chips=4096 TPU v32022.04 | 32.5 | — | |
| Megatron-Turing NLG# of Parameters (in billions)=530B, Accelerator chips=2240 A1002022.04 | 30.2 | — | |
| GPT-3# of Parameters (in billions)=175B, Accelerator chips=V1002022.04 | 21.3 | — |