# Multi-hop QA on MuSiQue (Violation Metrics)

Source: [Multi-hop QA on MuSiQue (Violation Metrics)](<https://www.sota2.com/research/sota/multi-hop-qa-on-musique-violation-metrics>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-10T09:36:15.765Z

Evaluation of model performance on the MuSiQue dataset for Multi-hop Question Answering, focusing specifically on violation and user satisfaction metrics (Avg Viol, VFR, Post-Res, USR).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Multi-hop QA |
| Dataset (including any stated split) | MuSiQue |
| Metric | Avg Violation |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 4 of 4 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Avg Violation](<https://www.sota2.com/research/sota/multi-hop-qa-on-musique-violation-metrics.md?metric=Avg%20Violation>): Lower is better; unit: SCALAR.

- [Violation Frequency Rate (VFR)](<https://www.sota2.com/research/sota/multi-hop-qa-on-musique-violation-metrics.md?metric=Violation%20Frequency%20Rate%20(VFR)>): Lower is better; unit: PERCENT.

- [Post-Response Score](<https://www.sota2.com/research/sota/multi-hop-qa-on-musique-violation-metrics.md?metric=Post-Response%20Score>): Higher is better; unit: SCALAR.

- [User Satisfaction Rate (USR)](<https://www.sota2.com/research/sota/multi-hop-qa-on-musique-violation-metrics.md?metric=User%20Satisfaction%20Rate%20(USR)>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Avg Violation | Result date | Report paper |
| --- | --- | --- | --- |
| SAVER | 0.83 | Not recorded | 69d8843d7d0172140ec77c26 |
| MAD | 2.16 | Not recorded | 69d8843d7d0172140ec77c26 |
| CoT | 2.91 | Not recorded | 69d8843d7d0172140ec77c26 |
| VANILLA LM | 3.25 | Not recorded | 69d8843d7d0172140ec77c26 |

### SAVER

Avg Violation: 0.83. Result ID: 69d8c48647f34ce924c208e1.

[Reporting paper](<https://www.sota2.com/research/paper/69d8843d7d0172140ec77c26.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d8843d7d0172140ec77c26.md>)

Recorded evaluation configuration: {"Model Scale":"LLaMA-3.1-8B"}. This record does not assert independent reproduction.

### MAD

Avg Violation: 2.16. Result ID: 69d8c48647f34ce924c208e0.

[Reporting paper](<https://www.sota2.com/research/paper/69d8843d7d0172140ec77c26.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75171e209131cea65abb.md>)

Recorded evaluation configuration: {"Model Scale":"LLaMA-3.1-8B"}. This record does not assert independent reproduction.

### CoT

Avg Violation: 2.91. Result ID: 69d8c48647f34ce924c208df.

[Reporting paper](<https://www.sota2.com/research/paper/69d8843d7d0172140ec77c26.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6e571e209131cea3f27b.md>)

Recorded evaluation configuration: {"Model Scale":"LLaMA-3.1-8B"}. This record does not assert independent reproduction.

### VANILLA LM

Avg Violation: 3.25. Result ID: 69d8c48647f34ce924c208de.

[Reporting paper](<https://www.sota2.com/research/paper/69d8843d7d0172140ec77c26.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Model Scale":"LLaMA-3.1-8B"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.