Constituency Parsing on Penn Treebank shortest 25% of samples < 128 tokens (test)
76.6Bracket PrecisionGPT-4
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| GPT-4Logit access=No, Few-shot prompting=4-shot2024.01 | 76.6 | 67.7 | 71.9 | 95.4 | 93.6 | 86 | |
| GPT-4 + SketchGCDLogit access=No, Few-shot prompting=4-shot, Constrained generator=LLaMA-7B2024.01 | 75.8 | 67.8 | 71.5 | 95.3 | 100 | 92.5 | |
| Claude 2.1Logit access=No, Few-shot prompting=4-shot2024.01 | 73.1 | 63.1 | 67.7 | 94.5 | 95.1 | 62.6 | |
| Claude 2.1 + SketchGCDLogit access=No, Few-shot prompting=4-shot, Constrained generator=LLaMA-7B2024.01 | 71.6 | 62.9 | 66.9 | 93.4 | 100 | 68.7 | |
| Claude-instant 1.2Logit access=No, Few-shot prompting=4-shot2024.01 | 71.3 | 59.1 | 64.7 | 89.6 | 91.7 | 56.6 | |
| GPT-3.5-Turbo + SketchGCDLogit access=No, Few-shot prompting=4-shot, Constrained generator=LLaMA-7B2024.01 | 68.7 | 56.6 | 62.1 | 92.6 | 100 | 81.5 | |
| GPT-3.5-TurboLogit access=No, Few-shot prompting=4-shot2024.01 | 68.2 | 55.5 | 61.2 | 93.1 | 91.7 | 76.9 | |
| Claude-instant 1.2 + SketchGCDLogit access=No, Few-shot prompting=4-shot, Constrained generator=LLaMA-7B2024.01 | 66.6 | 57.4 | 61.6 | 87.9 | 100 | 67.8 | |
| llama-2-70BLogit access=Yes, Few-shot prompting=4-shot2024.01 | 45.5 | 37.7 | 41.2 | 55.5 | 75.8 | 40.4 | |
| llama-2-70B + CDLogit access=Yes, Few-shot prompting=4-shot, Decoding=Constrained Decoding2024.01 | 39.8 | 35.6 | 37.6 | 53.8 | 100 | 47.6 | |
| llama-2-13BLogit access=Yes, Few-shot prompting=4-shot2024.01 | 33.4 | 22.4 | 26.8 | 29.3 | 95.5 | 38.5 | |
| llama-2-13B + CDLogit access=Yes, Few-shot prompting=4-shot, Decoding=Constrained Decoding2024.01 | 33.3 | 21.8 | 26.3 | 34 | 100 | 43.4 | |
| llama-2-7B + CDLogit access=Yes, Few-shot prompting=4-shot, Decoding=Constrained Decoding2024.01 | 28.5 | 16.5 | 20.9 | 13.8 | 100 | 35.1 | |
| llama-2-7BLogit access=Yes, Few-shot prompting=4-shot2024.01 | 23.1 | 10.4 | 14.3 | 14.9 | 93.2 | 32.1 |