Knowledge-Intensive Language Tasks on KILT (test)
21.6WoW F1 ScoreATLAS
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ATLASTraining setting=full train set2022.08 | 21.6 | 93.5 | 85.1 | 80.8 | 61.3 | 50.6 | 84 | 90.6 | — | — | — | — | — | — | |
| FID with RS2022.08 | 20.6 | 92.2 | 85.2 | 83.7 | 61.2 | 39.1 | 84.6 | — | — | — | — | — | — | — | |
| Re2G2022.08 | 18.9 | 89.6 | 87.7 | — | 51.7 | — | 76.3 | — | — | — | — | — | — | — | |
| UpperBoundTuning strategy=Full-parameter Fine-tuning2023.05 | 18.39 | 86.42 | 52.5 | 32.6 | 45.03 | 24.54 | — | — | 52.92 | 62.5 | 69.82 | 34.66 | 12.33 | 41.79 | |
| SEAL2022.08 | 18.3 | 89.5 | 83.6 | 74.6 | 53.7 | 40.5 | 70.9 | — | — | — | — | — | — | — | |
| UpperBoundTuning strategy=Parameter-efficient Tuning2023.05 | 17.92 | 88.2 | 49.96 | 33.65 | 42.6 | 23.84 | — | — | 50.86 | 61.77 | 69.14 | 33.71 | 11.8 | 41.22 | |
| PlugDTuning strategy=Full-parameter Fine-tuning, Backbone=T5-large, Pre-training=C4 unsupervised, Task-agnostic document representations=false2023.05 | 17.61 | 86.34 | 52.22 | 30.99 | 42.53 | 23.46 | — | — | 50.42 | 59.46 | 67.07 | 33.07 | 12.3 | 40.61 | |
| PlugD (w/o PT)Tuning strategy=Full-parameter Fine-tuning, Backbone=T5-large, Pre-training=none, Task-agnostic document representations=false2023.05 | 17.4 | 85.97 | 52.4 | 29.94 | 42.25 | 23.05 | — | — | 49.8 | 58.88 | 66.6 | 32.2 | 12.16 | 40.26 | |
| EmbRecyTuning strategy=Full-parameter Fine-tuning, Backbone=T5-large, Task-agnostic document representations=false2023.05 | 17.3 | 84.34 | 50.12 | 30.1 | 42.71 | 23.57 | — | — | 50.55 | 59.31 | 66.67 | 33.46 | 12.01 | 39.93 | |
| PlugDTuning strategy=Parameter-efficient Tuning, Backbone=T5-large, Pre-training=C4 unsupervised, Task-agnostic document representations=false2023.05 | 17.15 | 86.56 | 48.38 | 32.12 | 41.54 | 23.04 | — | — | 49.76 | 57.29 | 65.43 | 32.51 | 11.37 | 39.68 | |
| PlugD (w/o PT)Tuning strategy=Parameter-efficient Tuning, Backbone=T5-large, Pre-training=none, Task-agnostic document representations=false2023.05 | 17.07 | 86.33 | 48.26 | 30.64 | 40.24 | 22.04 | — | — | 47.72 | 57.67 | 64.91 | 31.44 | 11.67 | 39.24 | |
| EmbRecyTuning strategy=Parameter-efficient Tuning, Backbone=T5-large, Task-agnostic document representations=false2023.05 | 16.91 | 84.59 | 44.16 | 27.2 | 37.42 | 18.98 | — | — | 45.43 | 53.02 | 61.05 | 27.7 | 11.57 | 36.73 | |
| EmbRecyTuning strategy=Full-parameter Fine-tuning, Backbone=T5-large, Task-agnostic document representations=true2023.05 | 16.69 | 85.04 | 50.28 | 27.82 | 39.89 | 21.59 | — | — | 47.58 | 57.91 | 65.37 | 30.92 | 11.92 | 38.89 | |
| EmbRecyTuning strategy=Parameter-efficient Tuning, Backbone=T5-large, Task-agnostic document representations=true2023.05 | 16.65 | 84.59 | 41.22 | 20.76 | 32.13 | 18.18 | — | — | 40.17 | 47.59 | 55.37 | 26.79 | 11.92 | 34.13 | |
| ED2LMTuning strategy=Full-parameter Fine-tuning, Backbone=T5-large, Task-agnostic document representations=false2023.05 | 16.1 | 80.59 | 50.84 | 31.77 | 42.07 | 22.8 | — | — | 49.79 | 58.94 | 66.68 | 32.32 | 11.66 | 39.35 | |
| ED2LMTuning strategy=Full-parameter Fine-tuning, Backbone=T5-large, Task-agnostic document representations=true2023.05 | 15.83 | 84.17 | 48.04 | 24.19 | 40.84 | 21.61 | — | — | 48.57 | 57.05 | 64.92 | 30.7 | 11.94 | 37.96 | |
| Sphere2022.08 | 15.5 | 89 | 81.7 | 74.2 | 51.6 | 38.3 | 72.7 | — | — | — | — | — | — | — | |
| ATLASTraining setting=64-shot2022.08 | 15.5 | 87.1 | 58.9 | 74.9 | 43.6 | 34.7 | 76.4 | 66.5 | — | — | — | — | — | — | |
| ED2LMTuning strategy=Parameter-efficient Tuning, Backbone=T5-large, Task-agnostic document representations=false2023.05 | 15.24 | 83.13 | 46.34 | 31.15 | 38.34 | 19.84 | — | — | 46.04 | 53.84 | 62.05 | 28.63 | 11.24 | 37.39 | |
| ED2LMTuning strategy=Parameter-efficient Tuning, Backbone=T5-large, Task-agnostic document representations=true2023.05 | 15.2 | 81.81 | 44.78 | 27.09 | 35.62 | 19.07 | — | — | 44.18 | 52.01 | 59.82 | 27.81 | 11.01 | 35.82 | |
| Hindsight2022.04 | 0.192 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KGI2022.04 | 0.186 | 0.856 | 0.844 | 0.726 | 0.452 | — | 0.61 | — | — | — | — | — | — | — | |
| SEAL+FiDScoring=LM+FM, intersective=true2022.04 | 0.183 | 0.895 | 0.836 | 0.747 | 0.537 | 40.5 | 0.709 | — | — | — | — | — | — | — | |
| SEAL+FiDScoring=LM+FM2022.04 | 0.175 | 0.879 | 0.837 | 0.742 | 0.473 | 37.6 | 0.658 | — | — | — | — | — | — | — | |
| MT-DPR+FiD2022.04 | 0.157 | 0.89 | 0.825 | 0.717 | 0.499 | 36.9 | 0.71 | — | — | — | — | — | — | — | |
| MT-DPR-WEB+FiD2022.04 | 0.155 | 0.89 | 0.817 | 0.742 | 0.516 | 38.3 | 0.727 | — | — | — | — | — | — | — | |
| MT-DPR+BART2022.04 | 0.153 | 0.863 | — | 0.58 | 0.398 | 31.8 | 0.596 | — | — | — | — | — | — | — | |
| DPR+BART2022.04 | 0.152 | 0.867 | 0.592 | 0.304 | 0.413 | 25.2 | 0.586 | — | — | — | — | — | — | — | |
| RAG2022.04 | 0.131 | 0.863 | 0.592 | 0.447 | 0.444 | 27 | 0.713 | — | — | — | — | — | — | — | |
| GENRE2022.08 | — | — | — | — | — | — | — | 89.9 | — | — | — | — | — | — |