Question Answering on SearchQA (test)
75.1N-gram F1Cluster-Former
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Cluster-Formernumber of clusters=5122020.09 | 75.1 | — | 68 | |
| Cluster-Formernumber of clusters=2562020.09 | 74.5 | — | 67.5 | |
| Cluster-Formernumber of clusters=642020.09 | 74.2 | — | 67 | |
| Locality-Sensitive Hashing2020.09 | 73.5 | — | 66 | |
| Sliding Window2020.09 | 73.2 | — | 65.8 | |
| Sparse Attention2020.09 | 71.7 | — | 64.7 | |
| DECAPROP2018.11 | 70.8 | 62.2 | — | |
| DECAPROP2020.09 | 70.8 | — | 62.2 | |
| Multi-Passage BERTBackbone=BERT-Large2019.08 | 70.7 | — | 65.1 | |
| Multi passage BERT2020.09 | 70.7 | — | 65.1 | |
| Multi-Passage BERTBackbone=BERT-Base2019.08 | 70.6 | — | 65.2 | |
| BERT (Large)Backbone=BERT-Large2019.08 | 69.1 | — | — | |
| HAS-QA2019.08 | 68.7 | — | 62.7 | |
| BERT-RC2019.08 | 68.7 | — | 63.7 | |
| TraCRNet2019.08 | 65.1 | — | 52.9 | |
| OpenQA2019.08 | 64.5 | — | 58.8 | |
| DS-QA2020.09 | 64.5 | — | 58.8 | |
| Full Re-Rankercombination=three different re-rankers2017.11 | 63.2 | — | 57 | |
| Strength-Based Re-Rankervariant=Counting2017.11 | 61.6 | — | 54.2 | |
| Coverage-Based Re-Ranker2017.11 | 61.4 | — | 54.1 | |
| Baseline Re-Ranker (BM25)2017.11 | 60.7 | — | 51.9 | |
| Bi-Attention (200d Hybrid MRU-LSTM)Time=≈7 min2018.03 | 59.5 | 49.4 | — | |
| AMANDA2018.01 | 56.6 | 46.8 | — | |
| AMANDATime=≈8* min2018.03 | 56.6 | 46.8 | — | |
| AMANDA2018.11 | 56.6 | 46.8 | — | |
| AMANDAstatus=concurrent work2018.06 | 56.6 | — | 46.8 | |
| Strength-Based Re-Rankervariant=Probability2017.11 | 56.5 | — | 50.4 | |
| R³2017.11 | 55.3 | — | 49 | |
| R³2019.08 | 55.3 | — | 49 | |
| R32020.09 | 55.3 | — | 49 | |
| FH-RNN2018.11 | 53.4 | 46.8 | — | |
| Focused Hierarchical RNN2018.06 | 53.4 | — | 46.8 | |
| Bi-Attention (300d MRU)Time=≈2 min2018.03 | 53.3 | 46.8 | — | |
| LSTM2 + Pointer Softmax2018.06 | 51.9 | — | 41.7 | |
| Bi-Attention (150d BiLSTM)Time=≈7 min2018.03 | 49 | 38.6 | — | |
| DrQA2019.08 | 48.7 | — | 41.9 | |
| DrQA2020.09 | 48.7 | — | 41.9 | |
| LSTM1 + Pointer Softmax2018.06 | 48.7 | — | 39.7 | |
| AQA2017.11 | 47.4 | — | 40.5 | |
| Bi-Attention (300d LSTM)Time=≈6 min2018.03 | 46.4 | 38.2 | — | |
| AQA2018.06 | 45.6 | — | 38.7 | |
| Bi-Attention (300d Sim. MRU)Time=≈25 sec2018.03 | 43.1 | 42.9 | — | |
| BiDAF2017.11 | 34.6 | — | 28.6 | |
| ASR2018.01 | 22.8 | 41.3 | — | |
| ASR2018.03 | 22.8 | 41.3 | — | |
| ASR2018.11 | 22.8 | 41.3 | — | |
| ASR2018.06 | 22.8 | — | 41.3 | |
| Bi-Attention (No Encoder)Time=≈17 sec2018.03 | 12.3 | 18.9 | — | |
| HUMAN2018.06 | — | — | 43.9 | |
| Human Performance2017.11 | — | — | 43.9 | |
| TF-IDF max2018.03 | — | 12.7 | — | |
| TF-IDF max2018.11 | — | 12.7 | — | |
| TF-IDF Max2018.01 | — | 12.7 | — | |
| TF-IDF MAX2018.06 | — | — | 12.7 |