Syntactic Generalization on SG
85.5SG ScoreGiLT-GPT2
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GiLT-GPT22026.05 | 85.5 | — | — | — | — | — | — | — | |
| Post-GPT22026.05 | 84.6 | — | — | — | — | — | — | — | |
| DTGSyntactic Inductive Bias Type=Dependency-based2024.07 | 83.9 | — | — | — | — | — | — | — | |
| TGSyntactic Inductive Bias Type=Constituency-based2024.07 | 82.5 | — | — | — | — | — | — | — | |
| Bi-Up-Ex-MBranching strategy=Binary, Information flow direction=Up, Composition function type=External, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 82.4 | — | — | — | — | — | — | — | |
| PushdownSyntactic Inductive Bias Type=Constituency-based2024.07 | 82.3 | — | — | — | — | — | — | — | |
| TXLSyntactic Inductive Bias Type=Dependency-based, Model Variant=trans2024.07 | 81.1 | — | — | — | — | — | — | — | |
| Bi-Dn-Ex-MBranching strategy=Binary, Information flow direction=Down, Composition function type=External, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 80.9 | — | — | — | — | — | — | — | |
| Bi-Dn-In-MBranching strategy=Binary, Information flow direction=Down, Composition function type=Internal, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 80.9 | — | — | — | — | — | — | — | |
| Bi-Dn-Ex-NmBranching strategy=Binary, Information flow direction=Down, Composition function type=External, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 80.2 | — | — | — | — | — | — | — | |
| PLMSyntactic Inductive Bias Type=Constituency-based2024.07 | 80.2 | — | — | — | — | — | — | — | |
| Bi-Up-Ex-NmBranching strategy=Binary, Information flow direction=Up, Composition function type=External, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 80.1 | — | — | — | — | — | — | — | |
| Bi-Up-In-MBranching strategy=Binary, Information flow direction=Up, Composition function type=Internal, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 79.7 | — | — | — | — | — | — | — | |
| Nb-Up-In-NmBranching strategy=Non-binary, Information flow direction=Up, Composition function type=Internal, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 79.6 | — | — | — | — | — | — | — | |
| Bi-Dn-In-NmBranching strategy=Binary, Information flow direction=Down, Composition function type=Internal, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 79.4 | — | — | — | — | — | — | — | |
| pretrained GPT22026.05 | 79.4 | — | — | — | — | — | — | — | |
| Nb-Dn-In-NmBranching strategy=Non-binary, Information flow direction=Down, Composition function type=Internal, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 78.1 | — | — | — | — | — | — | — | |
| Bi-Up-In-NmBranching strategy=Binary, Information flow direction=Up, Composition function type=Internal, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 77.5 | — | — | — | — | — | — | — | |
| TXLSyntactic Inductive Bias Type=None, Model Variant=tokens2024.07 | 76.6 | — | — | — | — | — | — | — | |
| Nb-Dn-In-MBranching strategy=Non-binary, Information flow direction=Down, Composition function type=Internal, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 75.6 | — | — | — | — | — | — | — | |
| Nb-Up-In-MBranching strategy=Non-binary, Information flow direction=Up, Composition function type=Internal, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 73.4 | — | — | — | — | — | — | — | |
| GPT2-treebeam size=300, max consecutive opening actions=32025.06 | 73.1 | — | — | — | — | — | — | — | |
| GPT2-tokenbeam size=300, max consecutive opening actions=32025.06 | 64.1 | — | — | — | — | — | — | — | |
| Nb-Dn-Ex-MBranching strategy=Non-binary, Information flow direction=Down, Composition function type=External, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 51.9 | — | — | — | — | — | — | — | |
| Nb-Up-Ex-MBranching strategy=Non-binary, Information flow direction=Up, Composition function type=External, Composition objective=Modified, beam size=300, max consecutive opening actions=32025.06 | 51.8 | — | — | — | — | — | — | — | |
| Nb-Dn-Ex-NmBranching strategy=Non-binary, Information flow direction=Down, Composition function type=External, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 51.1 | — | — | — | — | — | — | — | |
| Nb-Up-Ex-NmBranching strategy=Non-binary, Information flow direction=Up, Composition function type=External, Composition objective=Normal, beam size=300, max consecutive opening actions=32025.06 | 40.8 | — | — | — | — | — | — | — | |
| GPSTmediumPre-training Corpus=OpenWebText2024.03 | — | 85.96 | 85.71 | 95.04 | 94.57 | 83.68 | 78.17 | 87.19 | |
| GPSTsmallPre-training Corpus=WikiText-1032024.03 | — | 59.65 | 73.21 | 87.1 | 97.83 | 57.89 | 64.78 | 73.41 | |
| GPSTsmallPre-training Corpus=OpenWebText2024.03 | — | 77.19 | 85.71 | 94.54 | 96.74 | 68.95 | 72.38 | 82.59 | |
| GPT2smallPre-training Corpus=WikiText-1032024.03 | — | 50.88 | 73.21 | 77.88 | 97.83 | 33.95 | 65.98 | 66.62 | |
| GPTmediumPre-training Corpus=OpenWebText2024.03 | — | 64.91 | 94.64 | 86.41 | 98.91 | 73.42 | 79.38 | 82.95 | |
| GPTsmallPre-training Corpus=OpenWebText2024.03 | — | 78.95 | 87.5 | 85.22 | 97.83 | 71.58 | 78.65 | 83.29 | |
| Pushdown LayersGold Trees=true2024.03 | — | 79 | 92 | 84.2 | 100 | 77.8 | 77.5 | 85.08 | |
| TGGold Trees=true2024.03 | — | 69.7 | 88.4 | 90.4 | 95.6 | 78.1 | 77.9 | 83.35 |