Open-domain Question Answering on SQuAD (test)
40.7AccuracyDenSPI + Sparc
Evaluation results
| Method | Links | |
|---|---|---|
| DenSPI + SparcCategory=Phrase Retrieval, Sparse=true, Storage (GB)=1547, #Q/sec (GPU)=2.1, #Q/sec (CPU)=1.72020.12 | 40.7 | |
| BERTSeriniCategory=Retriever-Reader, Sparse=true, Storage (GB)=21, #Q/sec (GPU)=2, #Q/sec (CPU)=0.42020.12 | 38.6 | |
| DensePhrasesCategory=Phrase Retrieval, Sparse=false, Storage (GB)=320, #Q/sec (GPU)=20.6, #Q/sec (CPU)=13.62020.12 | 38 | |
| DenSPICategory=Phrase Retrieval, Sparse=true, Storage (GB)=1200, #Q/sec (GPU)=2.9, #Q/sec (CPU)=2.42020.12 | 36.2 | |
| DrQACategory=Retriever-Reader, Sparse=true, Storage (GB)=26, #Q/sec (GPU)=1.8, #Q/sec (CPU)=0.62020.12 | 29.8 | |
| DPR-multiCategory=Retriever-Reader, Sparse=false, Storage (GB)=76, #Q/sec (GPU)=0.9, #Q/sec (CPU)=0.042020.12 | 24.1 | |
| ORQACategory=Retriever-Reader, Sparse=false, Storage (GB)=18, #Q/sec (GPU)=8.6, #Q/sec (CPU)=1.22020.12 | 20.2 |