Question Answering on HybridQA (dev)
68.4EM (Total)S3HQA
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| S3HQAFine-tuning=true2024.02 | 68.4 | — | — | 70.3 | 75.3 | 69.9 | 78.2 | 75.3 | |
| MAFIDFine-tuning=true2024.02 | 66.2 | — | — | 69.4 | 75.2 | 66.5 | 75.5 | 74.1 | |
| MITQAFine-tuning=true2024.02 | 65.5 | — | — | 68.1 | 73.3 | 66.7 | 75.6 | 72.7 | |
| POINTRFine-tuning=true2024.02 | 63.4 | — | — | 68.6 | 74.2 | 62.8 | 71.9 | 71 | |
| POINTR + MATEBackbone=MATE2021.09 | 63.4 | — | — | 68.6 | 74.2 | 62.8 | 71.9 | 71 | |
| POINTR + TAPASBackbone=TAPAS2021.09 | 63.3 | — | — | 68.1 | 73.9 | 62.9 | 72 | 70.8 | |
| POINTR + SATBackbone=SAT2021.09 | 61.2 | — | — | 66.5 | 71.8 | 60.3 | 69.2 | 68.7 | |
| POINTR + LINFORMERBackbone=LINFORMER2021.09 | 60.8 | — | — | 65.5 | 71.1 | 59.4 | 69 | 68.4 | |
| MuGER2Fine-tuning=true2024.02 | 57.1 | — | — | 60.9 | 69.2 | 56.9 | 68.9 | 67.3 | |
| HPROPROFine-tuning=false, Sampled cases=2002024.02 | 48 | — | — | 51.4 | 55.9 | 46.8 | 54.4 | 54.6 | |
| HYBRIDERFine-tuning=true2024.02 | 44 | — | — | 54.3 | 61.4 | 39.1 | 45.7 | 50.7 | |
| HYBRIDERT=0.82021.09 | 44 | — | — | 54.3 | 61.4 | 39.1 | 45.7 | 50.7 | |
| POINTR + TABLEETCBackbone=TABLEETC2021.09 | 36.1 | — | — | 36 | 42.4 | 37.8 | 45.3 | 42.9 | |
| Unsupervised-QGFine-tuning=false2024.02 | 25.7 | — | — | — | — | — | — | 30.5 | |
| GPT-4 End-to-End QA w. RetrieverFine-tuning=false, Sampled cases=2002024.02 | 24.5 | — | — | 50 | 61.8 | 11.1 | 13.3 | 30 | |
| Passage-Only2021.09 | 19.5 | — | — | 9.2 | 13.5 | 26.1 | 32.4 | 25.1 | |
| Table-Only2021.09 | 8.4 | — | — | 14.7 | 19.1 | 2.4 | 4.5 | 12.1 | |
| DOCHOPPERruntime=74.6/s2021.06 | — | 47.7 | 55 | — | — | — | — | — | |
| DOCHOPPERsparse_features=without2021.06 | — | 44.4 | 51.2 | — | — | — | — | — | |
| DOCHOPPERquery_update=without2021.06 | — | 44.2 | 50.9 | — | — | — | — | — | |
| DOCHOPPERscope=sentence-only2021.06 | — | 36.7 | 43.7 | — | — | — | — | — | |
| DOCHOPPERattention_steps=single-hop2021.06 | — | 27.8 | 34.1 | — | — | — | — | — | |
| DOCHOPPERinput_unit=cell2021.06 | — | 53.1 | 61.4 | — | — | — | — | — | |
| Hybriderruntime=5.1/s2021.06 | — | 44 | 50.7 | — | — | — | — | — | |
| HYBRIDERBackbone=BERT-Base2021.09 | — | 43.7 | 50.9 | — | — | — | — | — | |
| HYBRIDERBackbone=BERT-Large2021.09 | — | 44 | 50.7 | — | — | — | — | — | |
| MATEnote=failed to rerun2021.06 | — | 63.4 | 71 | — | — | — | — | — | |
| ReasonBERT2021.09 | — | 60.3 | 67.2 | — | — | — | — | — | |
| Retrieval + ETCretriever=BM25, runtime=8.3/s2021.06 | — | 37 | 43.5 | — | — | — | — | — | |
| Sequential (ETC)strategy=sequential reading, runtime=0.6±0.1/s2021.06 | — | 39.4 | 44.8 | — | — | — | — | — | |
| TAPAS+ROBERTa2021.09 | — | 57.3 | 64 | — | — | — | — | — |