Question Answering on HybridQA (test)
88.2EM (Total)Human
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Human2021.09 | 88.2 | — | — | — | — | — | — | 93.5 | |
| S3HQAFine-tuning=true2024.02 | 67.9 | — | — | 70.6 | 76.3 | 68.7 | 77.8 | 75.5 | |
| MAFIDFine-tuning=true2024.02 | 65.4 | — | — | 68.5 | 74.9 | 65.7 | 75.3 | 73.6 | |
| MITQA2021.12 | 64.3 | — | — | 68.5 | 74.4 | 64.3 | 73.3 | 71.9 | |
| MITQAFine-tuning=true2024.02 | 64.3 | — | — | 68.5 | 74.4 | 64.3 | 73.3 | 71.9 | |
| DEHGFine-tuning=true2024.02 | 63.9 | — | — | — | — | — | — | 75.5 | |
| POINTR + MATE2021.12 | 62.8 | — | — | 66.9 | 72.3 | 62.8 | 72.9 | 70.2 | |
| POINTRFine-tuning=true2024.02 | 62.8 | — | — | 66.9 | 72.3 | 62.8 | 71.9 | 70.2 | |
| POINTR + MATEBackbone=MATE2021.09 | 62.8 | — | — | 66.9 | 72.3 | 62.8 | 71.9 | 70.2 | |
| POINTR + TAPASBackbone=TAPAS2021.09 | 62.7 | — | — | 67.8 | 73.2 | 62 | 70.9 | 70 | |
| POINTR + LINFORMERBackbone=LINFORMER2021.09 | 60.2 | — | — | 66.1 | 71.7 | 58.9 | 67.8 | 67.6 | |
| POINTR + SATBackbone=SAT2021.09 | 60.1 | — | — | 64.6 | 70.1 | 59.6 | 68.5 | 67.4 | |
| MuGER2Fine-tuning=true2024.02 | 56.3 | — | — | 58.7 | 66.6 | 57.1 | 68.6 | 66.2 | |
| HPROPROFine-tuning=false2024.02 | 48.7 | — | — | 52.9 | 57.6 | 46.5 | 57.5 | 57.7 | |
| DocHopperFine-tuning=true2024.02 | 46.3 | — | — | — | — | — | — | 53.3 | |
| HYBRIDERtau=0.82021.12 | 43.8 | — | — | 56.2 | 63.3 | 37.5 | 44.4 | 50.6 | |
| HYBRIDERFine-tuning=true2024.02 | 43.8 | — | — | 56.2 | 63.3 | 37.5 | 44.4 | 50.6 | |
| HYBRIDERT=0.82021.09 | 43.8 | — | — | 56.2 | 63.3 | 37.5 | 44.4 | 50.6 | |
| POINTR + TABLEETCBackbone=TABLEETC2021.09 | 36.6 | — | — | 35.8 | 40.7 | 38.8 | 45.7 | 42.6 | |
| Passage-Only2021.12 | 19.1 | — | — | 8.9 | 13.8 | 25.5 | 32 | 25 | |
| Passage-Only2021.09 | 19.1 | — | — | 8.9 | 13.8 | 25.5 | 32 | 25 | |
| Table-Only2021.12 | 8.3 | — | — | 14.2 | 18.8 | 2.6 | 4.7 | 11.7 | |
| Table-Only2021.09 | 8.3 | — | — | 14.2 | 18.8 | 2.6 | 4.7 | 11.7 | |
| DOCHOPPERruntime=74.6/s2021.06 | — | 46.3 | 53.3 | — | — | — | — | — | |
| Hybriderruntime=5.1/s2021.06 | — | 43.8 | 50.6 | — | — | — | — | — | |
| HYBRIDERBackbone=BERT-Base2021.09 | — | 42.5 | 50.2 | — | — | — | — | — | |
| HYBRIDERBackbone=BERT-Large2021.09 | — | 43.8 | 50.6 | — | — | — | — | — | |
| MATEnote=failed to rerun2021.06 | — | 62.8 | 70.2 | — | — | — | — | — | |
| ReasonBERT2021.09 | — | 58 | 65.3 | — | — | — | — | — | |
| Retrieval + ETCretriever=BM25, runtime=8.3/s2021.06 | — | 34.1 | 40.3 | — | — | — | — | — | |
| Sequential (ETC)runtime=0.6±0.1/s2021.06 | — | 37 | 43 | — | — | — | — | — | |
| TAPAS+ROBERTa2021.09 | — | 56.1 | 63.3 | — | — | — | — | — |