Syntactic Evaluation on SyntaxGym Overall
82.5AccuracyTG‡
Evaluation results
| Method | Links | |
|---|---|---|
| TG‡Inference Complexity=Costly (external parser), Tokenization Scheme=Constituency-aligned2024.02 | 82.5 | |
| ChinchillaTraining Corpus Size=Large (250x-1000x)2024.02 | 79.7 | |
| GopherTraining Corpus Size=Large (250x-1000x)2024.02 | 79.5 | |
| GPT-2Training Corpus Size=Large (250x-1000x)2024.02 | 78.4 | |
| TPT [dep.]Inference Complexity=Standard, Supervision=Dependency distances, Tokenization Scheme=Dependency-aligned2024.02 | 77.1 | |
| TPT [cons.]Inference Complexity=Standard, Supervision=Constituency distances, Tokenization Scheme=Constituency-aligned2024.02 | 75.8 | |
| PLM-mask†Inference Complexity=Costly (beam search), Tokenization Scheme=Constituency-aligned2024.02 | 74.6 | |
| PLM+Inference Complexity=Costly (beam search), Tokenization Scheme=Constituency-aligned2024.02 | 73.2 | |
| TPT [bin.]Inference Complexity=Standard, Supervision=Binary constituency distances, Tokenization Scheme=Constituency-aligned2024.02 | 73 | |
| TPT [zero]Inference Complexity=Standard, Supervision=None (Transformer LM), Tokenization Scheme=Dependency-aligned2024.02 | 71.7 | |
| TPT [seq.]Inference Complexity=Standard, Supervision=Sequential distances, Tokenization Scheme=Dependency-aligned2024.02 | 70.1 | |
| TPT [rand.]Inference Complexity=Standard, Supervision=Random syntactic distances, Tokenization Scheme=Dependency-aligned2024.02 | 69 | |
| PLM-maskComparison Category=Comparable SLM2024.02 | 42.5 | |
| PLMComparison Category=Comparable SLM2024.02 | 42.2 |