Energy Consumption on SOTA Model Pre-training Corpora
24.1Representative Energy (MWh)GShard
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GShardDev.=Google, Params (B)=619.0, FLOPs (ZF)=133.3, L=48, W=2048, GPUs=100, Tokens/Epoch=1e14, C_epoch (FLOPs)=2.57e23, M_proxy (Proxy)=5.03e212026.06 | 24.1 | 0.04 | 1.72 | 1.45 | 69.08 | |
| T5Dev.=Google, Params (B)=11.0, FLOPs (ZF)=40.5, L=24, W=1024, GPUs=100, Tokens/Epoch=1e14, C_epoch (FLOPs)=3.40e22, M_proxy (Proxy)=1.26e212026.06 | 85.7 | 0.02 | 2.65 | 0.85 | 106.47 | |
| SwitchDev.=Google, Params (B)=1500.0, FLOPs (ZF)=82.2, L=32, W=2048, GPUs=100, Tokens/Epoch=1e14, C_epoch (FLOPs)=1.71e23, M_proxy (Proxy)=3.36e212026.06 | 179 | 0.03 | 8.4 | 1.29 | 337.34 | |
| MeenaDev.=Google, Params (B)=2.6, FLOPs (ZF)=112.0, L=24, W=1024, GPUs=100, Tokens/Epoch=1e14, C_epoch (FLOPs)=3.21e22, M_proxy (Proxy)=6.29e202026.06 | 232 | 0.02 | 6.9 | 0.79 | 276.94 | |
| GPT-3Dev.=OpenAI, Params (B)=175.0, FLOPs (ZF)=314.0, L=96, W=12288, GPUs=100, Tokens/Epoch=1e14, C_epoch (FLOPs)=1.81e25, M_proxy (Proxy)=2.42e232026.06 | 1,287 | 0.12 | 17.33 | 4.88 | 696.13 |