# Question Answering on PopQA (Accuracy)

Source: [Question Answering on PopQA (Accuracy)](<https://www.sota2.com/research/sota/question-answering-on-popqa-accuracy>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-08T07:19:07.100Z

Evaluation of model performance on the PopQA dataset for Question Answering, measured using the Accuracy metric.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Question Answering |
| Dataset (including any stated split) | PopQA |
| Metric | Accuracy |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 158 of 158 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Accuracy](<https://www.sota2.com/research/sota/question-answering-on-popqa-accuracy.md?metric=Accuracy>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Accuracy | Result date | Report paper |
| --- | --- | --- | --- |
| NWCAD | 87.12 | Not recorded | 69e704667d0172140ec7eb3c |
| NWCAD | 85.69 | Not recorded | 69e704667d0172140ec7eb3c |
| AdaCAD | 85.62 | Not recorded | 69e704667d0172140ec7eb3c |
| With-context | 85.12 | Not recorded | 69e704667d0172140ec7eb3c |
| AdaCAD | 84.94 | Not recorded | 69e704667d0172140ec7eb3c |
| NWCAD | 84.12 | Not recorded | 69e704667d0172140ec7eb3c |
| CoCoA | 84.12 | Not recorded | 69e704667d0172140ec7eb3c |
| With-context | 83.81 | Not recorded | 69e704667d0172140ec7eb3c |
| AdaCAD | 82.69 | Not recorded | 69e704667d0172140ec7eb3c |
| With-context | 82.38 | Not recorded | 69e704667d0172140ec7eb3c |
| CoCoA | 80.94 | Not recorded | 69e704667d0172140ec7eb3c |
| KARLA | 80.91 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| KARLA | 80.63 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| CAD | 80.31 | Not recorded | 69e704667d0172140ec7eb3c |
| KARLA | 78.98 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| KARLA | 78.56 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| CAD | 78.44 | Not recorded | 69e704667d0172140ec7eb3c |
| NWCAD | 77.88 | Not recorded | 69e704667d0172140ec7eb3c |
| NWCAD | 75.29 | Not recorded | 69e704667d0172140ec7eb3c |
| AdaCAD | 74.62 | Not recorded | 69e704667d0172140ec7eb3c |
| Neuro-RIT | 74.29 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| CoCoA | 74 | Not recorded | 69e704667d0172140ec7eb3c |
| AdaCAD | 73.4 | Not recorded | 69e704667d0172140ec7eb3c |
| NWCAD | 72.8 | Not recorded | 69e704667d0172140ec7eb3c |
| CoCoA | 72.8 | Not recorded | 69e704667d0172140ec7eb3c |
| CAD | 72.5 | Not recorded | 69e704667d0172140ec7eb3c |
| CoCoA | 72.38 | Not recorded | 69e704667d0172140ec7eb3c |
| AdaCAD | 71.8 | Not recorded | 69e704667d0172140ec7eb3c |
| PA-RAG | 71.08 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| Neuro-RIT + Reranker | 70.65 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| No compression | 70 | Not recorded | 68fa77ee1e209131cea76195 |
| No compression | 70 | Not recorded | 68fa77ee1e209131cea76195 |
| OSCAR-llama | 70 | Not recorded | 68fa77ee1e209131cea76195 |
| RALM + Reranker | 69.05 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| Provence | 69 | Not recorded | 68fa77ee1e209131cea76195 |
| With-context | 68.88 | Not recorded | 69e704667d0172140ec7eb3c |
| CoCoA | 68.8 | Not recorded | 69e704667d0172140ec7eb3c |
| OSCAR-llama | 68 | Not recorded | 68fa77ee1e209131cea76195 |
| OSCAR-8-Layers | 68 | Not recorded | 68fa77ee1e209131cea76195 |
| OSCAR-llama | 68 | Not recorded | 68fa77ee1e209131cea76195 |
| Neuro-RIT | 67.37 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RECOMP | 67 | Not recorded | 68fa77ee1e209131cea76195 |
| OSCAR-8-Layers | 67 | Not recorded | 68fa77ee1e209131cea76195 |
| KARLA-no-desc | 66.98 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| KARLA-no-desc | 66.38 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| OSCAR-5-Layers | 66 | Not recorded | 68fa77ee1e209131cea76195 |
| KARLA-no-desc | 65.76 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| PA-RAG | 65.68 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| CAD | 65.6 | Not recorded | 69e704667d0172140ec7eb3c |
| With-context | 65.2 | Not recorded | 69e704667d0172140ec7eb3c |
| InstructRAG | 64.6 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| PA-RAG | 64.37 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| Neuro-RIT | 64.15 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| PISCO | 64 | Not recorded | 68fa77ee1e209131cea76195 |
| No compression | 64 | Not recorded | 68fa77ee1e209131cea76195 |
| CAD | 63.6 | Not recorded | 69e704667d0172140ec7eb3c |
| Neuro-RIT | 63.28 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| Neuro-RIT + RankCoT | 63.19 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RALM | 62.78 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| With-context | 62.4 | Not recorded | 69e704667d0172140ec7eb3c |
| InstructRAG | 62.24 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RALM | 60.85 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RALM + RankCoT | 60.83 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RALM | 60.37 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| InstructRAG | 60.25 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RALM | 59.8 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| No compression | 59 | Not recorded | 68fa77ee1e209131cea76195 |
| OSCAR-5-Layers | 59 | Not recorded | 68fa77ee1e209131cea76195 |
| CAD | 59 | Not recorded | 69e704667d0172140ec7eb3c |
| 1-hop graph RAG | 58.68 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| 1-hop graph RAG | 56.17 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| KARLA-no-desc | 55.97 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| 1-hop graph RAG | 55.02 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| RetRobust | 54.92 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| 1-hop graph RAG | 54.45 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| RetRobust | 53.24 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| RetRobust | 53.18 | Not recorded | 69cf49c10636e5bf4afcdc22 |
| LMLM | 52 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| LMLM | 50.8 | Not recorded | 6a3e07c3206542251d7cb6c7 |
| State-SFT & RL‡ | 42.8 | Not recorded | 6a41fc14206542251d7ccb08 |

### NWCAD

Accuracy: 87.12. Result ID: 69e72d8c47f34ce924cd3067.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### NWCAD

Accuracy: 85.69. Result ID: 69e72d8c47f34ce924cd306d.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### AdaCAD

Accuracy: 85.62. Result ID: 69e72d8c47f34ce924cd3065.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78411e209131cea776f7.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### With-context

Accuracy: 85.12. Result ID: 69e72d8c47f34ce924cd3069.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### AdaCAD

Accuracy: 84.94. Result ID: 69e72d8c47f34ce924cd306b.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78411e209131cea776f7.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### NWCAD

Accuracy: 84.12. Result ID: 69e72d8c47f34ce924cd3061.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### CoCoA

Accuracy: 84.12. Result ID: 69e72d8c47f34ce924cd3066.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7add1e209131cea8594e.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### With-context

Accuracy: 83.81. Result ID: 69e72d8c47f34ce924cd305d.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### AdaCAD

Accuracy: 82.69. Result ID: 69e72d8c47f34ce924cd305f.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78411e209131cea776f7.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### With-context

Accuracy: 82.38. Result ID: 69e72d8c47f34ce924cd3063.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### CoCoA

Accuracy: 80.94. Result ID: 69e72d8c47f34ce924cd3060.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7add1e209131cea8594e.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### KARLA

Accuracy: 80.91. Result ID: 6a3e37a5fbc373592c6026b3.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 4B","Setup":"KARLA"}. This record does not assert independent reproduction.

### KARLA

Accuracy: 80.63. Result ID: 6a3e37a5fbc373592c6026b9.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 8B","Setup":"KARLA"}. This record does not assert independent reproduction.

### CAD

Accuracy: 80.31. Result ID: 69e72d8c47f34ce924cd306a.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6e1e209131cea45813.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### KARLA

Accuracy: 78.98. Result ID: 6a3e37a5fbc373592c6026ad.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 1.7B","Setup":"KARLA"}. This record does not assert independent reproduction.

### KARLA

Accuracy: 78.56. Result ID: 6a3e37a5fbc373592c6026a7.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 0.6B","Setup":"KARLA"}. This record does not assert independent reproduction.

### CAD

Accuracy: 78.44. Result ID: 69e72d8c47f34ce924cd305e.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6e1e209131cea45813.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### NWCAD

Accuracy: 77.88. Result ID: 69e72d8c47f34ce924cd30c1.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### NWCAD

Accuracy: 75.29. Result ID: 69e72d8c47f34ce924cd30c7.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### AdaCAD

Accuracy: 74.62. Result ID: 69e72d8c47f34ce924cd30bf.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78411e209131cea776f7.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### Neuro-RIT

Accuracy: 74.29. Result ID: 69cf858047f34ce924b9a7fa.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

Recorded evaluation configuration: {"Ranking Strategy":"Reranker"}. This record does not assert independent reproduction.

### CoCoA

Accuracy: 74. Result ID: 69e72d8c47f34ce924cd306c.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7add1e209131cea8594e.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### AdaCAD

Accuracy: 73.4. Result ID: 69e72d8c47f34ce924cd30c5.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78411e209131cea776f7.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### NWCAD

Accuracy: 72.8. Result ID: 69e72d8c47f34ce924cd30bb.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### CoCoA

Accuracy: 72.8. Result ID: 69e72d8c47f34ce924cd30c6.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7add1e209131cea8594e.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### CAD

Accuracy: 72.5. Result ID: 69e72d8c47f34ce924cd3064.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6e1e209131cea45813.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### CoCoA

Accuracy: 72.38. Result ID: 69e72d8c47f34ce924cd30c0.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7add1e209131cea8594e.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### AdaCAD

Accuracy: 71.8. Result ID: 69e72d8c47f34ce924cd30b9.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78411e209131cea776f7.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### PA-RAG

Accuracy: 71.08. Result ID: 69cf858047f34ce924b9a7f7.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76481e209131cea6c21e.md>)

Recorded evaluation configuration: {"Ranking Strategy":"Reranker"}. This record does not assert independent reproduction.

### Neuro-RIT + Reranker

Accuracy: 70.65. Result ID: 69cf858047f34ce924b9a7c1.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

Recorded evaluation configuration: {"Generator":"Mistral-7B-Instruct-v0.2"}. This record does not assert independent reproduction.

### No compression

Accuracy: 70. Result ID: 69a93bd46dbb7075b9528e27.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### No compression

Accuracy: 70. Result ID: 69a93bd46dbb7075b9528e33.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-24B"}. This record does not assert independent reproduction.

### OSCAR-llama

Accuracy: 70. Result ID: 69a93bd46dbb7075b9528e34.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-24B"}. This record does not assert independent reproduction.

### RALM + Reranker

Accuracy: 69.05. Result ID: 69cf858047f34ce924b9a7be.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6d021e209131cea37450.md>)

[Implementation](<https://github.com/IBM/kgi-slot-filling/tree/re2g>)

Recorded evaluation configuration: {"Generator":"Mistral-7B-Instruct-v0.2"}. This record does not assert independent reproduction.

### Provence

Accuracy: 69. Result ID: 69a93bd46dbb7075b9528e29.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76cd1e209131cea6f29e.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### With-context

Accuracy: 68.88. Result ID: 69e72d8c47f34ce924cd30bd.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### CoCoA

Accuracy: 68.8. Result ID: 69e72d8c47f34ce924cd30ba.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7add1e209131cea8594e.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### OSCAR-llama

Accuracy: 68. Result ID: 69a93bd46dbb7075b9528e2b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### OSCAR-8-Layers

Accuracy: 68. Result ID: 69a93bd46dbb7075b9528e2d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### OSCAR-llama

Accuracy: 68. Result ID: 69a93bd46dbb7075b9528e32.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-7B"}. This record does not assert independent reproduction.

### Neuro-RIT

Accuracy: 67.37. Result ID: 69cf858047f34ce924b9a7f9.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

Recorded evaluation configuration: {"Ranking Strategy":"None"}. This record does not assert independent reproduction.

### RECOMP

Accuracy: 67. Result ID: 69a93bd46dbb7075b9528e28.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70c81e209131cea4d0a5.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### OSCAR-8-Layers

Accuracy: 67. Result ID: 69a93bd46dbb7075b9528e31.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-7B"}. This record does not assert independent reproduction.

### KARLA-no-desc

Accuracy: 66.98. Result ID: 6a3e37a5fbc373592c6026b4.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 4B","Setup":"KARLA-no-desc"}. This record does not assert independent reproduction.

### KARLA-no-desc

Accuracy: 66.38. Result ID: 6a3e37a5fbc373592c6026ba.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 8B","Setup":"KARLA-no-desc"}. This record does not assert independent reproduction.

### OSCAR-5-Layers

Accuracy: 66. Result ID: 69a93bd46dbb7075b9528e2c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### KARLA-no-desc

Accuracy: 65.76. Result ID: 6a3e37a5fbc373592c6026ae.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 1.7B","Setup":"KARLA-no-desc"}. This record does not assert independent reproduction.

### PA-RAG

Accuracy: 65.68. Result ID: 69cf858047f34ce924b9a7f8.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76481e209131cea6c21e.md>)

Recorded evaluation configuration: {"Ranking Strategy":"RankCoT"}. This record does not assert independent reproduction.

### CAD

Accuracy: 65.6. Result ID: 69e72d8c47f34ce924cd30c4.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6e1e209131cea45813.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### With-context

Accuracy: 65.2. Result ID: 69e72d8c47f34ce924cd30c3.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Ministral-3-8B"}. This record does not assert independent reproduction.

### InstructRAG

Accuracy: 64.6. Result ID: 69cf858047f34ce924b9a7f4.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa773e1e209131cea71ed6.md>)

Recorded evaluation configuration: {"Ranking Strategy":"Reranker"}. This record does not assert independent reproduction.

### PA-RAG

Accuracy: 64.37. Result ID: 69cf858047f34ce924b9a7f6.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76481e209131cea6c21e.md>)

Recorded evaluation configuration: {"Ranking Strategy":"None"}. This record does not assert independent reproduction.

### Neuro-RIT

Accuracy: 64.15. Result ID: 69cf858047f34ce924b9a7c0.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

Recorded evaluation configuration: {"Generator":"Mistral-7B-Instruct-v0.2"}. This record does not assert independent reproduction.

### PISCO

Accuracy: 64. Result ID: 69a93bd46dbb7075b9528e2a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76b41e209131cea6e946.md>)

[Implementation](<https://www.sota2.com/huggingface.co/naver/pisco-mistral>)

Recorded evaluation configuration: {"Backbone":"Mistral-7B"}. This record does not assert independent reproduction.

### No compression

Accuracy: 64. Result ID: 69a93bd46dbb7075b9528e30.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-7B"}. This record does not assert independent reproduction.

### CAD

Accuracy: 63.6. Result ID: 69e72d8c47f34ce924cd30b8.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6e1e209131cea45813.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### Neuro-RIT

Accuracy: 63.28. Result ID: 69cf858047f34ce924b9a7fb.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

Recorded evaluation configuration: {"Ranking Strategy":"RankCoT"}. This record does not assert independent reproduction.

### Neuro-RIT + RankCoT

Accuracy: 63.19. Result ID: 69cf858047f34ce924b9a7c2.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

Recorded evaluation configuration: {"Generator":"Mistral-7B-Instruct-v0.2"}. This record does not assert independent reproduction.

### RALM

Accuracy: 62.78. Result ID: 69cf858047f34ce924b9a7bd.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Generator":"Mistral-7B-Instruct-v0.2"}. This record does not assert independent reproduction.

### With-context

Accuracy: 62.4. Result ID: 69e72d8c47f34ce924cd30b7.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-8B"}. This record does not assert independent reproduction.

### InstructRAG

Accuracy: 62.24. Result ID: 69cf858047f34ce924b9a7f5.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa773e1e209131cea71ed6.md>)

Recorded evaluation configuration: {"Ranking Strategy":"RankCoT"}. This record does not assert independent reproduction.

### RALM

Accuracy: 60.85. Result ID: 69cf858047f34ce924b9a7ed.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Ranking Strategy":"None"}. This record does not assert independent reproduction.

### RALM + RankCoT

Accuracy: 60.83. Result ID: 69cf858047f34ce924b9a7bf.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa771d1e209131cea711b5.md>)

[Implementation](<https://github.com/NEUIR/RankCoT>)

Recorded evaluation configuration: {"Generator":"Mistral-7B-Instruct-v0.2"}. This record does not assert independent reproduction.

### RALM

Accuracy: 60.37. Result ID: 69cf858047f34ce924b9a7ee.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Ranking Strategy":"Reranker"}. This record does not assert independent reproduction.

### InstructRAG

Accuracy: 60.25. Result ID: 69cf858047f34ce924b9a7f3.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa773e1e209131cea71ed6.md>)

Recorded evaluation configuration: {"Ranking Strategy":"None"}. This record does not assert independent reproduction.

### RALM

Accuracy: 59.8. Result ID: 69cf858047f34ce924b9a7ef.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa69b91e209131cea23c26.md>)

[Implementation](<https://github.com/huggingface/transformers/blob/master/examples/rag/>)

Recorded evaluation configuration: {"Ranking Strategy":"RankCoT"}. This record does not assert independent reproduction.

### No compression

Accuracy: 59. Result ID: 69a93bd46dbb7075b9528e2e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Llama-1B"}. This record does not assert independent reproduction.

### OSCAR-5-Layers

Accuracy: 59. Result ID: 69a93bd46dbb7075b9528e2f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa77ee1e209131cea76195.md>)

Recorded evaluation configuration: {"Backbone":"Llama-1B"}. This record does not assert independent reproduction.

### CAD

Accuracy: 59. Result ID: 69e72d8c47f34ce924cd30be.

[Reporting paper](<https://www.sota2.com/research/paper/69e704667d0172140ec7eb3c.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6f6e1e209131cea45813.md>)

Recorded evaluation configuration: {"Model":"Llama-3.1-70B"}. This record does not assert independent reproduction.

### 1-hop graph RAG

Accuracy: 58.68. Result ID: 6a3e37a5fbc373592c6026bd.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 8B","Setup":"1-hop graph RAG"}. This record does not assert independent reproduction.

### 1-hop graph RAG

Accuracy: 56.17. Result ID: 6a3e37a5fbc373592c6026b7.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 4B","Setup":"1-hop graph RAG"}. This record does not assert independent reproduction.

### KARLA-no-desc

Accuracy: 55.97. Result ID: 6a3e37a5fbc373592c6026a8.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 0.6B","Setup":"KARLA-no-desc"}. This record does not assert independent reproduction.

### 1-hop graph RAG

Accuracy: 55.02. Result ID: 6a3e37a5fbc373592c6026b1.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 1.7B","Setup":"1-hop graph RAG"}. This record does not assert independent reproduction.

### RetRobust

Accuracy: 54.92. Result ID: 69cf858047f34ce924b9a7f1.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73121e209131cea5a44d.md>)

[Implementation](<https://github.com/oriyor/ret-robust>)

Recorded evaluation configuration: {"Ranking Strategy":"Reranker"}. This record does not assert independent reproduction.

### 1-hop graph RAG

Accuracy: 54.45. Result ID: 6a3e37a5fbc373592c6026ab.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

Recorded evaluation configuration: {"Model":"Qwen 0.6B","Setup":"1-hop graph RAG"}. This record does not assert independent reproduction.

### RetRobust

Accuracy: 53.24. Result ID: 69cf858047f34ce924b9a7f0.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73121e209131cea5a44d.md>)

[Implementation](<https://github.com/oriyor/ret-robust>)

Recorded evaluation configuration: {"Ranking Strategy":"None"}. This record does not assert independent reproduction.

### RetRobust

Accuracy: 53.18. Result ID: 69cf858047f34ce924b9a7f2.

[Reporting paper](<https://www.sota2.com/research/paper/69cf49c10636e5bf4afcdc22.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73121e209131cea5a44d.md>)

[Implementation](<https://github.com/oriyor/ret-robust>)

Recorded evaluation configuration: {"Ranking Strategy":"RankCoT"}. This record does not assert independent reproduction.

### LMLM

Accuracy: 52. Result ID: 6a3e37a5fbc373592c6026bf.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bd71e209131cea8ad9f.md>)

Recorded evaluation configuration: {"Model":"LLAMA2-382M","Setup":"LMLM"}. This record does not assert independent reproduction.

### LMLM

Accuracy: 50.8. Result ID: 6a3e37a5fbc373592c6026c0.

[Reporting paper](<https://www.sota2.com/research/paper/6a3e07c3206542251d7cb6c7.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7bd71e209131cea8ad9f.md>)

Recorded evaluation configuration: {"Model":"GPT2-774M","Setup":"LMLM"}. This record does not assert independent reproduction.

### State-SFT & RL‡

Accuracy: 42.8. Result ID: 6a421073fbc373592c865975.

[Reporting paper](<https://www.sota2.com/research/paper/6a41fc14206542251d7ccb08.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a41fc14206542251d7ccb08.md>)

Recorded evaluation configuration: {"Backbone":"Youtu-LLM-2B-Base","Framework":"WM-AMT","Optimization Stage":"RL stage"}. This record does not assert independent reproduction.

Page 1 of 2. This is an index page, not the full collection.

- [Next index page](<https://www.sota2.com/research/sota/question-answering-on-popqa-accuracy.md?metric=Accuracy&page=2>): Follow this link to discover more pages.
