Language Modeling on BigPatent
21.91PerplexityGPT-2 Large + DP-SGD (teacher)
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-2 Large + DP-SGD (teacher)ε=1.02026.04 | 21.91 | |
| DP-OPDε=2.02026.04 | 30.63 | |
| DistilGPT-2 + DISTILDP (DP synthetic text)ε=2.02026.04 | 32.43 | |
| DistilGPT-2 + DPKD (off-policy DP KD)ε=2.02026.04 | 41.2 | |
| DistilGPT-2 + DP-SGD (student-only)ε=2.02026.04 | 41.8 |