Event Relation Extraction on MAVEN-ERE 1.0 (test)
56.8Micro F1ROBERTa-Large
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| ROBERTa-Largemode=fully fine-tuned2023.10 | 56.8 | 6.4 | |
| GPT-4prompting=w. all logical constraints2023.10 | 37.3 | 8.3 | |
| GPT-4prompting=w. retrieved logical constraints2023.10 | 33.5 | 28.8 | |
| GPT-4prompting=CoT w. logical constraints2023.10 | 32.3 | 13.7 | |
| GPT-4prompting=vanilla CoT2023.10 | 30.3 | 36.7 | |
| GPT-4prompting=vanilla ICL2023.10 | 29.3 | 50.7 | |
| DavinciBackbone=text-davinci-003, prompting=w. retrieved logical constraints2023.10 | 27.8 | 30.8 | |
| DavinciBackbone=text-davinci-003, prompting=w. all logical constraints2023.10 | 27 | 25.6 | |
| Llama2-FTBackbone=Llama2-13B, Finetuning=LLM-ERL, prompting=CoT w. logical constraints2023.10 | 26.4 | 15.7 | |
| TurboBackbone=gpt-3.5-turbo, prompting=CoT w. logical constraints2023.10 | 25.3 | 37.9 | |
| DavinciBackbone=text-davinci-003, prompting=CoT w. logical constraints2023.10 | 24.8 | 5.5 | |
| TurboBackbone=gpt-3.5-turbo, prompting=w. retrieved logical constraints2023.10 | 22.3 | 30.2 | |
| Llama2-FTBackbone=Llama2-13B, Finetuning=LLM-ERL, prompting=vanilla CoT2023.10 | 22.1 | 42.9 | |
| DavinciBackbone=text-davinci-003, prompting=vanilla ICL2023.10 | 21.6 | 49.1 | |
| Llama2Backbone=Llama2-13B, prompting=CoT w. logical constraints2023.10 | 21.5 | 18.9 | |
| TurboBackbone=gpt-3.5-turbo, prompting=w. all logical constraints2023.10 | 20.8 | 30.9 | |
| DavinciBackbone=text-davinci-003, prompting=vanilla CoT2023.10 | 20.5 | 60.5 | |
| Llama2-FTBackbone=Llama2-13B, Finetuning=LLM-ERL, prompting=w. all logical constraints2023.10 | 20.2 | 28.7 | |
| Llama2Backbone=Llama2-13B, prompting=w. all logical constraints2023.10 | 19.5 | 34.6 | |
| Llama2-FTBackbone=Llama2-13B, Finetuning=LLM-ERL, prompting=vanilla ICL2023.10 | 19 | 45.8 | |
| TurboBackbone=gpt-3.5-turbo, prompting=vanilla CoT2023.10 | 18.8 | 49.3 | |
| Llama2-FTBackbone=Llama2-13B, Finetuning=LLM-ERL, prompting=w. retrieved logical constraints2023.10 | 18.7 | 34.2 | |
| Llama2Backbone=Llama2-13B, prompting=w. retrieved logical constraints2023.10 | 18.3 | 38.2 | |
| TurboBackbone=gpt-3.5-turbo, prompting=vanilla ICL2023.10 | 18 | 53.3 | |
| Vicuna-FTBackbone=Vicuna-13B, Finetuning=LLM-ERL, prompting=CoT w. logical constraints2023.10 | 18 | 6 | |
| Llama2Backbone=Llama2-13B, prompting=vanilla CoT2023.10 | 17.8 | 58.4 | |
| ROBERTa-Largemode=one-shot2023.10 | 17.4 | 54.8 | |
| GPT-4prompting=w. post-processing2023.10 | 17 | 0 | |
| Llama2Backbone=Llama2-13B, prompting=vanilla ICL2023.10 | 17 | 54.6 | |
| Vicuna-FTBackbone=Vicuna-13B, Finetuning=LLM-ERL, prompting=w. all logical constraints2023.10 | 16.3 | 8.7 | |
| Vicuna-FTBackbone=Vicuna-13B, Finetuning=LLM-ERL, prompting=w. retrieved logical constraints2023.10 | 16.1 | 19 | |
| Vicuna-FTBackbone=Vicuna-13B, Finetuning=LLM-ERL, prompting=vanilla CoT2023.10 | 15.8 | 17.8 | |
| VicunaBackbone=Vicuna-13B, prompting=w. retrieved logical constraints2023.10 | 15.7 | 33.2 | |
| Vicuna-FTBackbone=Vicuna-13B, Finetuning=LLM-ERL, prompting=vanilla ICL2023.10 | 15.3 | 21.2 | |
| VicunaBackbone=Vicuna-13B, prompting=w. all logical constraints2023.10 | 15.2 | 37.6 | |
| VicunaBackbone=Vicuna-13B, prompting=CoT w. logical constraints2023.10 | 14.9 | 21.7 | |
| DavinciBackbone=text-davinci-003, prompting=w. post-processing2023.10 | 14.8 | 0 | |
| TurboBackbone=gpt-3.5-turbo, prompting=w. post-processing2023.10 | 14 | 0 | |
| VicunaBackbone=Vicuna-13B, prompting=vanilla ICL2023.10 | 13.8 | 25.4 | |
| Llama2Backbone=Llama2-13B, prompting=w. post-processing2023.10 | 12 | 0 | |
| VicunaBackbone=Vicuna-13B, prompting=vanilla CoT2023.10 | 11.6 | 47.4 | |
| Vicuna-FTBackbone=Vicuna-13B, Finetuning=LLM-ERL, prompting=w. post-processing2023.10 | 11 | 0 | |
| Llama2-FTBackbone=Llama2-13B, Finetuning=LLM-ERL, prompting=w. post-processing2023.10 | 11 | 0 | |
| VicunaBackbone=Vicuna-13B, prompting=w. post-processing2023.10 | 9.8 | 0 |