Few-shot Language Understanding on GPT-3 Evaluation Suite
48.1AccuracyGPT-3 (Original)
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-3 (Original)Model size=1.3B, Train tokens=300B, Batch size=512, Bsz warmup=8B, LR=2e-4, min LR=2e-5, LR warmup=375M, LR decay=260B, decay style=cosine, SLW=N/A2021.08 | 48.1 | |
| GPT-3 (SLW 8x Bsz)Model size=1.3B, Train tokens=300B, Batch size=4K, Bsz warmup=N/A, LR=8e-4, min LR=2e-5, LR warmup=375M, LR decay=260B, decay style=cosine, SLW=11K steps2021.08 | 45.3 | |
| GPT-3 (Baseline repro)Model size=1.3B, Train tokens=300B, Batch size=512, Bsz warmup=8B, LR=2e-4, min LR=2e-5, LR warmup=375M, LR decay=260B, decay style=cosine, SLW=N/A2021.08 | 44.8 |