# Role-playing agent evaluation on LLM Court 5 legal scenarios 1.0 (test)

Source: [Role-playing agent evaluation on LLM Court 5 legal scenarios 1.0 (test)](<https://www.sota2.com/research/sota/role-playing-agent-evaluation-on-llm-court-5-legal-scenarios-1-0-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-14T12:31:59.744Z

Evaluation of agent role-playing capabilities using the LLM Court 5 legal scenarios 1.0 test set, measured by metrics such as QS\_d\_BRF and Retry Rate.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Role-playing agent evaluation |
| Dataset (including any stated split) | LLM Court 5 legal scenarios 1.0 (test) |
| Metric | QS d BRF Score |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 7 of 7 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [QS d BRF Score](<https://www.sota2.com/research/sota/role-playing-agent-evaluation-on-llm-court-5-legal-scenarios-1-0-test.md?metric=QS%20d%20BRF%20Score>): Higher is better; unit: PERCENT.

- [Retry Rate](<https://www.sota2.com/research/sota/role-playing-agent-evaluation-on-llm-court-5-legal-scenarios-1-0-test.md?metric=Retry%20Rate>): Lower is better; unit: SCALAR.

## Recorded results

| Method | QS d BRF Score | Result date | Report paper |
| --- | --- | --- | --- |
| Llama-3.1-8B | 92.5 | Not recorded | 69ddca7e7d0172140ec79fb6 |
| Gemma-2-9B-it | 89.3 | Not recorded | 69ddca7e7d0172140ec79fb6 |
| Qwen-3-8B | 89.3 | Not recorded | 69ddca7e7d0172140ec79fb6 |
| Gemma-3-12B-it | 82.3 | Not recorded | 69ddca7e7d0172140ec79fb6 |
| Hermes-3-8B | 81 | Not recorded | 69ddca7e7d0172140ec79fb6 |
| Phi-4-14B | 76 | Not recorded | 69ddca7e7d0172140ec79fb6 |
| Qwen-3-14B | 73 | Not recorded | 69ddca7e7d0172140ec79fb6 |

### Llama-3.1-8B

QS d BRF Score: 92.5. Result ID: 69de321d47f34ce924c7045f.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"4-bit"}. This record does not assert independent reproduction.

### Gemma-2-9B-it

QS d BRF Score: 89.3. Result ID: 69de321d47f34ce924c70460.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"4-bit"}. This record does not assert independent reproduction.

### Qwen-3-8B

QS d BRF Score: 89.3. Result ID: 69de321d47f34ce924c70461.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"4-bit"}. This record does not assert independent reproduction.

### Gemma-3-12B-it

QS d BRF Score: 82.3. Result ID: 69de321d47f34ce924c70462.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"4-bit"}. This record does not assert independent reproduction.

### Hermes-3-8B

QS d BRF Score: 81. Result ID: 69de321d47f34ce924c70463.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"5-bit"}. This record does not assert independent reproduction.

### Phi-4-14B

QS d BRF Score: 76. Result ID: 69de321d47f34ce924c70464.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"4-bit"}. This record does not assert independent reproduction.

### Qwen-3-14B

QS d BRF Score: 73. Result ID: 69de321d47f34ce924c70465.

[Reporting paper](<https://www.sota2.com/research/paper/69ddca7e7d0172140ec79fb6.md>)

Recorded evaluation configuration: {"Quantization":"4-bit"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
