# Open-Domain Question Answering on Natural Questions (NQ) (EM, Q/s) Source: [Open-Domain Question Answering on Natural Questions (NQ) (EM, Q/s)]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:41:35.441Z Evaluation of model performance on the Natural Questions (NQ) dataset for Open-Domain Question Answering, measured using Exact Match (EM) and throughput (Q/s). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Open-Domain Question Answering | | Dataset (including any stated split) | NQ (Natural Questions) | | Metric | EM | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 33 of 33 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [EM](): Higher is better; unit: PERCENT. - [Throughput (Q/s)](): Higher is better; unit: SCALAR. ## Recorded results | Method | EM | Result date | Report paper | | --- | --- | --- | --- | | FiD-large | 51.4 | Not recorded | 68fa6dc31e209131cea3bead | | FiD-base | 48.2 | Not recorded | 68fa6dc31e209131cea3bead | | RePAQ rerank | 45.7 | Not recorded | 68fa6dc31e209131cea3bead | | QAMAT | 44.7 | Not recorded | 68fa6dc31e209131cea3bead | | RAG | 44.5 | Not recorded | 68fa6dc31e209131cea3bead | | EMAT-FKSV | 44.3 | Not recorded | 68fa6dc31e209131cea3bead | | EMAT-SKSV | 43.3 | Not recorded | 68fa6dc31e209131cea3bead | | RePAQ-xlarge | 41.5 | Not recorded | 68fa6dc31e209131cea3bead | | DPR | 41.5 | Not recorded | 68fa6dc31e209131cea3bead | | RePAQ-large | 41.2 | Not recorded | 68fa6dc31e209131cea3bead | | DensePhrases | 40.9 | Not recorded | 68fa6dc31e209131cea3bead | | RePAQ-base | 40.9 | Not recorded | 68fa6dc31e209131cea3bead | | REALM | 40.4 | Not recorded | 68fa6dc31e209131cea3bead | | RA-ISF | 40.2 | Not recorded | 68fa738b1e209131cea5d1d4 | | SKRknn | 33.8 | Not recorded | 68fa738b1e209131cea5d1d4 | | IRCoT | 32.9 | Not recorded | 68fa738b1e209131cea5d1d4 | | T5-11B | 32.6 | Not recorded | 68fa6dc31e209131cea3bead | | Least-to-most | 32.5 | Not recorded | 68fa738b1e209131cea5d1d4 | | RAG | 31.7 | Not recorded | 68fa738b1e209131cea5d1d4 | | RA-ISF | 31.3 | Not recorded | 68fa738b1e209131cea5d1d4 | | T5-3B | 30.4 | Not recorded | 68fa6dc31e209131cea3bead | | Direct | 29.2 | Not recorded | 68fa738b1e209131cea5d1d4 | | Self-RAG13B | 28.4 | Not recorded | 68fa738b1e209131cea5d1d4 | | T5-large | 27.6 | Not recorded | 68fa6dc31e209131cea3bead | | Dense Retriever | 26.7 | Not recorded | 68fa6dc31e209131cea3bead | | BART-large | 26.5 | Not recorded | 68fa6dc31e209131cea3bead | | T5-base | 25.8 | Not recorded | 68fa6dc31e209131cea3bead | | REPLUG | 23.8 | Not recorded | 68fa738b1e209131cea5d1d4 | | IRCoT | 23.4 | Not recorded | 68fa738b1e209131cea5d1d4 | | Least-to-most | 22.8 | Not recorded | 68fa738b1e209131cea5d1d4 | | RAG | 21.6 | Not recorded | 68fa738b1e209131cea5d1d4 | | SKRknn | 20.8 | Not recorded | 68fa738b1e209131cea5d1d4 | | Vanilla LM | 17.4 | Not recorded | 68fa738b1e209131cea5d1d4 | ### FiD-large EM: 51.4. Result ID: 698fdacb3a6c314849fc592d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented"}. This record does not assert independent reproduction. ### FiD-base EM: 48.2. Result ID: 698fdacb3a6c314849fc592c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented","Backbone":"T5-base"}. This record does not assert independent reproduction. ### RePAQ rerank EM: 45.7. Result ID: 698fdacb3a6c314849fc592a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented"}. This record does not assert independent reproduction. ### QAMAT EM: 44.7. Result ID: 698fdacb3a6c314849fc5929. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented","Hardware":"32 TPU-v3"}. This record does not assert independent reproduction. ### RAG EM: 44.5. Result ID: 698fdacb3a6c314849fc592b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented"}. This record does not assert independent reproduction. ### EMAT-FKSV EM: 44.3. Result ID: 698fdacb3a6c314849fc592e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access Delay Setting":"Fast Key, Slow Value (FKSV)","Hardware":"A100 GPU","Backbone":"T5-base"}. This record does not assert independent reproduction. ### EMAT-SKSV EM: 43.3. Result ID: 698fdacb3a6c314849fc592f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access Delay Setting":"Slow Key, Slow Value (SKSV)","Hardware":"A100 GPU","Backbone":"T5-base"}. This record does not assert independent reproduction. ### RePAQ-xlarge EM: 41.5. Result ID: 698fdacb3a6c314849fc5926. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-only"}. This record does not assert independent reproduction. ### DPR EM: 41.5. Result ID: 698fdacb3a6c314849fc5928. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented"}. This record does not assert independent reproduction. ### RePAQ-large EM: 41.2. Result ID: 698fdacb3a6c314849fc5925. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-only"}. This record does not assert independent reproduction. ### DensePhrases EM: 40.9. Result ID: 698fdacb3a6c314849fc5923. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Type":"Retrieval-only"}. This record does not assert independent reproduction. ### RePAQ-base EM: 40.9. Result ID: 698fdacb3a6c314849fc5924. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-only"}. This record does not assert independent reproduction. ### REALM EM: 40.4. Result ID: 698fdacb3a6c314849fc5927. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-augmented"}. This record does not assert independent reproduction. ### RA-ISF EM: 40.2. Result ID: 6992ef833a6c314849127844. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"GPT-3.5","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### SKRknn EM: 33.8. Result ID: 6992ef833a6c314849127843. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"GPT-3.5","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### IRCoT EM: 32.9. Result ID: 6992ef833a6c314849127841. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"GPT-3.5","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### T5-11B EM: 32.6. Result ID: 698fdacb3a6c314849fc5920. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Parametric"}. This record does not assert independent reproduction. ### Least-to-most EM: 32.5. Result ID: 6992ef833a6c314849127840. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"GPT-3.5","Retrieval Configuration":"Without Retrieval"}. This record does not assert independent reproduction. ### RAG EM: 31.7. Result ID: 6992ef833a6c314849127842. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"GPT-3.5","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### RA-ISF EM: 31.3. Result ID: 6992ef833a6c31484912784c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### T5-3B EM: 30.4. Result ID: 698fdacb3a6c314849fc591f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Parametric"}. This record does not assert independent reproduction. ### Direct EM: 29.2. Result ID: 6992ef833a6c31484912783f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"GPT-3.5","Retrieval Configuration":"Without Retrieval"}. This record does not assert independent reproduction. ### Self-RAG13B EM: 28.4. Result ID: 6992ef833a6c31484912784b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### T5-large EM: 27.6. Result ID: 698fdacb3a6c314849fc591e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Parametric"}. This record does not assert independent reproduction. ### Dense Retriever EM: 26.7. Result ID: 698fdacb3a6c314849fc5922. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Retrieval-only"}. This record does not assert independent reproduction. ### BART-large EM: 26.5. Result ID: 698fdacb3a6c314849fc5921. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model Type":"Parametric"}. This record does not assert independent reproduction. ### T5-base EM: 25.8. Result ID: 698fdacb3a6c314849fc591d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model Type":"Parametric"}. This record does not assert independent reproduction. ### REPLUG EM: 23.8. Result ID: 6992ef833a6c31484912784a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### IRCoT EM: 23.4. Result ID: 6992ef833a6c314849127847. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### Least-to-most EM: 22.8. Result ID: 6992ef833a6c314849127846. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"Without Retrieval"}. This record does not assert independent reproduction. ### RAG EM: 21.6. Result ID: 6992ef833a6c314849127848. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### SKRknn EM: 20.8. Result ID: 6992ef833a6c314849127849. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"With Retrieval"}. This record does not assert independent reproduction. ### Vanilla LM EM: 17.4. Result ID: 6992ef833a6c314849127845. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Model":"Llama-2-13b","Retrieval Configuration":"Without Retrieval"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.