Zero-shot Evaluation on GPT-3 Evaluation Suite (LAMBADA, TriviaQA, WebQs, PIQA, RACE-h, BoolQ)
44.4Overall AccuracyGPT-3 1.3B (Original)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| GPT-3 1.3B (Original)Case=Original [6], Model size=1.3B, Train tokens=300B, Batch size=512, Bsz warmup=8B, LR=2e-4, min LR=2e-5, LR warmup=375M, LR decay=260B, decay style=cosine, SLW=N/A, Evaluation Protocol=Zero-shot2021.08 | 44.4 | 63.6 | 19.7 | 4.63 | 75.1 | 40.9 | 62.4 | |
| GPT-3 1.3B (SLW)Case=8x Bsz, Model size=1.3B, Train tokens=300B, Batch size=4K, Bsz warmup=N/A, LR=8e-4, min LR=2e-5, LR warmup=375M, LR decay=260B, decay style=cosine, SLW=11K steps, Evaluation Protocol=Zero-shot2021.08 | 41.9 | 65 | 11.3 | 2.36 | 73.8 | 37.1 | 61.8 | |
| GPT-3 1.3B (Baseline repro)Case=repro, Model size=1.3B, Train tokens=300B, Batch size=512, Bsz warmup=8B, LR=2e-4, min LR=2e-5, LR warmup=375M, LR decay=260B, decay style=cosine, SLW=N/A, Evaluation Protocol=Zero-shot2021.08 | 41.6 | 63.7 | 10.1 | 3.25 | 73.4 | 35.6 | 63.4 |