Statement Extraction (Direct) on SemTabNet (test)
0Invalid OutputSEM-T5-small
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SEM-T5-smallContext Length=5122024.06 | 0 | 98.14 | 0.04 | 1.82 | 60.65 | 0.62 | |
| SEM-T5-baseContext Length=5122024.06 | 0 | 97.86 | 0.06 | 2.09 | 68.62 | 4.46 | |
| SEM-T5-largeContext Length=5122024.06 | 0 | 98.18 | 0.02 | 1.8 | 67.41 | 4.23 | |
| SEM-T5-3bContext Length=5122024.06 | 0 | 97.98 | 0.01 | 2.01 | 70.06 | 3.47 | |
| SEM-T5-smallContext Length=10242024.06 | 0 | 92.93 | 0.14 | 6.93 | 70.35 | 2.98 | |
| SEM-T5-baseContext Length=10242024.06 | 0 | 88.42 | 0.22 | 11.35 | 76.99 | 11.11 | |
| SEM-T5-largeContext Length=10242024.06 | 0 | 89.34 | 0.21 | 10.45 | 76.59 | 6.06 | |
| Falcon-40bContext Length=2048, Evaluation Protocol=1-shot In-Context2024.06 | 12.59 | 45.69 | 28.77 | 25.54 | 17.94 | 0.15 | |
| Llama-2-13bContext Length=4096, Evaluation Protocol=1-shot In-Context2024.06 | 17.93 | 79.95 | 5.78 | 14.27 | 37.94 | 5.29 | |
| Mixtral-8x7bContext Length=8192, Evaluation Protocol=1-shot In-Context2024.06 | 19.2 | 56.39 | 18.06 | 25.56 | 6.51 | 21.07 | |
| Mistral-7bContext Length=8192, Evaluation Protocol=1-shot In-Context2024.06 | 21.92 | 53.37 | 18.76 | 27.87 | 16.92 | 11.57 | |
| Llama-2-70bContext Length=4096, Evaluation Protocol=1-shot In-Context2024.06 | 24.82 | 89.65 | 2.56 | 7.79 | 3.18 | 6.31 |