Language Modeling on WebText (test)
0.87Diversity (Div)GPT-2 (MLE)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-2 (MLE)Model Size=Small, Objective=MLE, Sampling=Unbiased2023.05 | 0.87 | 21.45 | 0.9 | 0.555 | |
| GPT-2 (MLE)Model Size=Medium, Objective=MLE, Sampling=Unbiased2023.05 | 0.87 | 15.92 | 0.88 | 0.56 | |
| GPT-2 (MLE)Model Size=Large, Objective=MLE, Sampling=Unbiased2023.05 | 0.87 | 14.13 | 0.81 | 0.57 | |
| GPT-2 (MIXCE)Model Size=Small, Objective=MIXCE, Sampling=Unbiased2023.05 | 0.85 | 21.69 | 0.92 | 0.565 | |
| MLEModel Size=Medium, best p=0.93, decoding=top-p sampling2023.05 | 0.85 | — | 0.95 | 0.592 | |
| HumanSampling=Unbiased2023.05 | 0.84 | — | 1 | 0.633 | |
| Humandecoding=unbiased sampling2023.05 | 0.84 | — | 1 | 0.633 | |
| MLEModel Size=Small, best p=0.93, decoding=top-p sampling2023.05 | 0.84 | — | 0.94 | 0.58 | |
| MIXCEModel Size=Small, best p=0.99, decoding=top-p sampling2023.05 | 0.84 | — | 0.93 | 0.571 | |
| MLEModel Size=Large, best p=0.95, decoding=top-p sampling2023.05 | 0.84 | — | 0.87 | 0.593 | |
| GPT-2 (MIXCE)Model Size=Medium, Objective=MIXCE, Sampling=Unbiased2023.05 | 0.83 | 16.51 | 0.93 | 0.585 | |
| GPT-2 (MIXCE)Model Size=Large, Objective=MIXCE, Sampling=Unbiased2023.05 | 0.82 | 14.66 | 0.94 | 0.592 | |
| MIXCEModel Size=Medium, best p=0.99, decoding=top-p sampling2023.05 | 0.81 | — | 0.93 | 0.594 | |
| MIXCEModel Size=Large, best p=0.99, decoding=top-p sampling2023.05 | 0.81 | — | 0.94 | 0.601 |