Answer Selection on WikiQA (test)
0.928MAPASR
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ASRBackbone=RoBERTa Large, K=32021.07 | 0.928 | 0.9399 | — | — | — | — | 0.8971 | |
| Garg et al.Backbone=RoBERTa Large2021.07 | 0.92 | 0.933 | — | — | — | — | — | |
| Our RerankerBackbone=RoBERTa Large2021.07 | 0.9151 | 0.9266 | — | — | — | — | 0.8724 | |
| KGATBackbone=RoBERTa Large, K=22021.07 | 0.9094 | 0.9218 | — | — | — | — | 0.8642 | |
| TANDA ELECTRABase Model=ELECTRA-Base, Intermediate transfer step=ASNQ2022.05 | 0.902 | 0.914 | — | — | — | — | 0.856 | |
| ASRBackbone=RoBERTa Base, k=32021.07 | 0.9014 | 0.9123 | — | — | — | — | 0.8436 | |
| MASR-FPBackbone=RoBERTa Base, k=32021.07 | 0.8998 | 0.9113 | — | — | — | — | 0.8436 | |
| KGATBackbone=RoBERTa Base, k=22021.07 | 0.8991 | 0.912 | — | — | — | — | 0.8436 | |
| RoBERTaBase + SSP (DPC)fine-tuning context=true2023.09 | 0.899 | 0.911 | — | — | — | — | 0.852 | |
| RoBERTaBase + SSP (DPC)CTX=true2023.09 | 0.899 | 0.911 | — | — | — | — | 0.852 | |
| Joint Model PairwiseBackbone=RoBERTa Base, k=32021.07 | 0.8927 | 0.9045 | — | — | — | — | 0.8272 | |
| MASR-FBackbone=RoBERTa Base, k=32021.07 | 0.8918 | 0.9031 | — | — | — | — | 0.8272 | |
| MASRBackbone=RoBERTa Base, k=32021.07 | 0.8891 | 0.9017 | — | — | — | — | 0.823 | |
| Reranker by Garg et al., 2020Backbone=RoBERTa Base2021.07 | 0.889 | 0.901 | — | — | — | — | — | |
| ELECTRABase + SSP (SDC)fine-tuning context=true2023.09 | 0.887 | 0.9 | — | — | — | — | 0.829 | |
| ELECTRABase + SSP (SDC)CTX=true2023.09 | 0.887 | 0.9 | — | — | — | — | 0.829 | |
| RoBERTa + SSPBase Model=RoBERTa-Base, Pre-training objective=SSP2022.05 | 0.887 | 0.899 | — | — | — | — | 0.829 | |
| Our RerankerBackbone=RoBERTa Base2021.07 | 0.886 | 0.8983 | — | — | — | — | 0.8189 | |
| ELECTRA + SSPBase Model=ELECTRA-Base, Pre-training objective=SSP2022.05 | 0.886 | 0.9 | — | — | — | — | 0.825 | |
| Joint MSPP IEkArchitecture=RoBERTa-Base, Pre-training task=MSPP, Evaluation Protocol=Joint model, Prediction Head=IEk, Fine-tuning=FT IEk2022.05 | 0.885 | 0.89 | — | — | — | — | 0.827 | |
| TANDA RoBERTaBase Model=RoBERTa-Base, Intermediate transfer step=ASNQ2022.05 | 0.885 | 0.899 | — | — | — | — | 0.83 | |
| RoBERTaBase + SSP (ALL)fine-tuning context=true2023.09 | 0.882 | 0.896 | — | — | — | — | 0.824 | |
| RoBERTaBase + SSP (ALL)CTX=true2023.09 | 0.882 | 0.896 | — | — | — | — | 0.824 | |
| RoBERTa + AllBase Model=RoBERTa-Base, Pre-training objective=SSP+SP+PSD2022.05 | 0.882 | 0.895 | — | — | — | — | 0.825 | |
| ELECTRA + SPBase Model=ELECTRA-Base, Pre-training objective=SP2022.05 | 0.881 | 0.895 | — | — | — | — | 0.818 | |
| ELECTRABase + SSP (DPC)fine-tuning context=true2023.09 | 0.88 | 0.892 | — | — | — | — | 0.813 | |
| ELECTRABase + SSP (DPC)CTX=true2023.09 | 0.88 | 0.892 | — | — | — | — | 0.813 | |
| Joint MSPP AEkArchitecture=RoBERTa-Base, Pre-training task=MSPP, Evaluation Protocol=Joint model, Prediction Head=AEk, Fine-tuning=FT AEk2022.05 | 0.879 | 0.89 | — | — | — | — | 0.819 | |
| RoBERTaBase + SSP (SDC)fine-tuning context=true2023.09 | 0.878 | 0.892 | — | — | — | — | 0.818 | |
| RoBERTaBase + SSP (DSLC)fine-tuning context=true2023.09 | 0.878 | 0.892 | — | — | — | — | 0.816 | |
| RoBERTaBase + SSP (SDC)CTX=true2023.09 | 0.878 | 0.892 | — | — | — | — | 0.818 | |
| RoBERTaBase + SSP (DSLC)CTX=true2023.09 | 0.878 | 0.892 | — | — | — | — | 0.816 | |
| RoBERTa + SPBase Model=RoBERTa-Base, Pre-training objective=SP2022.05 | 0.877 | 0.889 | — | — | — | — | 0.81 | |
| ELECTRABase + SSP (ALL)fine-tuning context=true2023.09 | 0.875 | 0.891 | — | — | — | — | 0.815 | |
| ELECTRABase + SSP (ALL)CTX=true2023.09 | 0.875 | 0.891 | — | — | — | — | 0.815 | |
| ELECTRA + AllBase Model=ELECTRA-Base, Pre-training objective=SSP+SP+PSD2022.05 | 0.873 | 0.887 | — | — | — | — | 0.808 | |
| DeBERTaV3Base + SSP (ALL)fine-tuning context=true2023.09 | 0.872 | 0.886 | — | — | — | — | 0.809 | |
| DeBERTaBasefine-tuning context=false2023.09 | 0.871 | 0.886 | — | — | — | — | 0.81 | |
| ELECTRABase + SSP (DSLC)fine-tuning context=true2023.09 | 0.87 | 0.885 | — | — | — | — | 0.797 | |
| ELECTRABase + SSP (DSLC)CTX=true2023.09 | 0.87 | 0.885 | — | — | — | — | 0.797 | |
| DeBERTaV3Basefine-tuning context=false2023.09 | 0.867 | 0.88 | — | — | — | — | 0.793 | |
| RoBERTa + PSDBase Model=RoBERTa-Base, Pre-training objective=PSD2022.05 | 0.864 | 0.88 | — | — | — | — | 0.805 | |
| RoBERTa-BaseBase Model=RoBERTa-Base2022.05 | 0.858 | 0.872 | — | — | — | — | 0.783 | |
| ELECTRABasefine-tuning context=false2023.09 | 0.857 | 0.871 | — | — | — | — | 0.785 | |
| ELECTRABaseCTX=false2023.09 | 0.857 | 0.871 | — | — | — | — | 0.785 | |
| ELECTRA + PSDBase Model=ELECTRA-Base, Pre-training objective=PSD2022.05 | 0.856 | 0.873 | — | — | — | — | 0.786 | |
| Joint Model Multi-classifierBackbone=RoBERTa Base, k=52021.07 | 0.8542 | 0.8684 | — | — | — | — | 0.7819 | |
| Pairwise RoBERTa-BaseArchitecture=RoBERTa-Base, Evaluation Protocol=Pairwise cross-encoder2022.05 | 0.853 | 0.865 | — | — | — | — | 0.771 | |
| RoBERTaBasefine-tuning context=false2023.09 | 0.851 | 0.865 | — | — | — | — | 0.772 | |
| RoBERTaBaseCTX=false2023.09 | 0.851 | 0.865 | — | — | — | — | 0.772 | |
| ELECTRA-BaseBase Model=ELECTRA-Base2022.05 | 0.85 | 0.865 | — | — | — | — | 0.771 | |
| DeBERTaBasefine-tuning context=true2023.09 | 0.848 | 0.864 | — | — | — | — | 0.771 | |
| RoBERTaBasefine-tuning context=true2023.09 | 0.844 | 0.86 | — | — | — | — | 0.77 | |
| RoBERTaBaseCTX=true2023.09 | 0.844 | 0.86 | — | — | — | — | 0.77 | |
| DeBERTaBase + SSP (ALL)fine-tuning context=true2023.09 | 0.838 | 0.852 | — | — | — | — | 0.755 | |
| Comp-Clip + LM + LC + TLLearning approach=Pointwise, Implementation=Ours2019.05 | 0.834 | 0.848 | — | — | — | — | — | |
| ELECTRABasefine-tuning context=true2023.09 | 0.831 | 0.844 | — | — | — | — | 0.738 | |
| ELECTRABaseCTX=true2023.09 | 0.831 | 0.844 | — | — | — | — | 0.738 | |
| Comp-Clip + LM + LC + TLLearning approach=Listwise, Implementation=Ours2019.05 | 0.83 | 0.841 | — | — | — | — | — | |
| DeBERTaV3Basefine-tuning context=true2023.09 | 0.779 | 0.793 | — | — | — | — | 0.667 | |
| Comp-Clip + LM + LCLearning approach=Pointwise, Implementation=Ours2019.05 | 0.764 | 0.784 | — | — | — | — | — | |
| Comp-Clip + LM + LCLearning approach=Listwise, Implementation=Ours2019.05 | 0.759 | 0.772 | — | — | — | — | — | |
| DEIM2022.03 | 0.755 | 0.775 | — | — | — | — | — | |
| DITM2022.03 | 0.752 | 0.77 | — | — | — | — | — | |
| Comp-Clip + LMLearning approach=Listwise, Implementation=Ours2019.05 | 0.748 | 0.768 | — | — | — | — | — | |
| Comp-Clip + LMLearning approach=Pointwise, Implementation=Ours2019.05 | 0.746 | 0.762 | — | — | — | — | — | |
| RE2loss=point-wise binary classification2019.08 | 0.7452 | 0.7618 | — | — | — | — | — | |
| RE22022.03 | 0.745 | 0.762 | — | — | — | — | — | |
| CA2019.08 | 0.7433 | 0.7545 | — | — | — | — | — | |
| Wang and Jiang2017.02 | 0.743 | 0.755 | — | — | — | — | — | |
| Wang and Jiang, 20172018.08 | 0.743 | 0.755 | — | — | — | — | — | |
| HCRNloss=pairwise ranking2019.08 | 0.743 | 0.756 | — | — | — | — | — | |
| HCRN2022.03 | 0.743 | 0.762 | — | — | — | — | — | |
| Wang et al.paper_tag=2016a2017.02 | 0.734 | 0.742 | — | — | — | — | — | |
| IWAN2019.08 | 0.733 | 0.75 | — | — | — | — | — | |
| CNN-RNF-LSTMvariant=CNN, Recurrent Filter=LSTM2018.08 | 0.729 | 0.747 | — | — | — | — | — | |
| CNN-RNF-GRUvariant=CNN, Recurrent Filter=GRU2018.08 | 0.726 | 0.738 | — | — | — | — | — | |
| BiMPM2017.02 | 0.718 | 0.731 | — | — | — | — | — | |
| BiMPM2019.08 | 0.718 | 0.731 | — | — | — | — | — | |
| Comp-ClipImplementation=Author's implementation (*)2019.05 | 0.718 | 0.732 | — | — | — | — | — | |
| BIMPM2022.03 | 0.718 | 0.731 | — | — | — | — | — | |
| IWAN + SCARNNImplementation=Author's implementation (*)2019.05 | 0.716 | 0.722 | — | — | — | — | — | |
| Comp-ClipLearning approach=Pointwise, Implementation=Ours2019.05 | 0.714 | 0.732 | — | — | — | — | — | |
| HYPERQA (This work)number of parameters=90K, runtime per epoch=2s2017.07 | 0.712 | 0.727 | — | — | — | — | — | |
| GRU-maxpoolvariant=RNN, pooling=max-pooling2018.08 | 0.712 | 0.724 | — | — | — | — | — | |
| He and Lin2017.02 | 0.709 | 0.723 | — | — | — | — | — | |
| Comp-ClipLearning approach=Listwise, Implementation=Ours2019.05 | 0.708 | 0.725 | — | — | — | — | — | |
| Key-Value Memory Networkhops=2, representation=Window-Level, window_size=72016.06 | 0.7069 | 0.7265 | — | — | — | — | — | |
| KVMN2019.08 | 0.7069 | 0.7265 | — | — | — | — | — | |
| Wang et al.paper_tag=2016d2017.02 | 0.706 | 0.723 | — | — | — | — | — | |
| L.D.C.2016.06 | 0.7058 | 0.7226 | — | — | — | — | — | |
| Lexical Decomposition and Composition Model2016.02 | 0.7058 | 0.7226 | — | — | — | — | — | |
| Rao et al.2017.02 | 0.701 | 0.718 | — | — | — | — | — | |
| Rank MP-CNN (Rao et al.)number of parameters=10.0M, runtime per epoch=33s2017.07 | 0.701 | 0.718 | — | — | — | — | — | |
| LSTM-maxpoolvariant=RNN, pooling=max-pooling2018.08 | 0.701 | 0.711 | — | — | — | — | — | |
| Rao et al., 20162018.08 | 0.701 | 0.718 | — | — | — | — | — | |
| Compare-AggregateImplementation=Author's implementation (*)2019.05 | 0.699 | 0.708 | — | — | — | — | — | |
| MP-CNN (He et al.)number of parameters=10.0M, runtime per epoch=35s2017.07 | 0.693 | 0.709 | — | — | — | — | — | |
| Attentive CNN2016.06 | 0.6921 | 0.7108 | — | — | — | — | — | |
| Yin et al.Model architecture=Attention-based CNN2016.02 | 0.6921 | 0.7108 | — | — | — | — | — |