# Agentic Text Synthesis on RAVEL

Source: [Agentic Text Synthesis on RAVEL](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-04T10:20:43.488Z

Evaluation of models performing Agentic Text Synthesis on the RAVEL dataset, assessed using metrics related to success rate (S%), trajectory efficiency (ηtraj %), reference adherence (ρref%), gain (δgain), and judge scores.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Agentic Text Synthesis |
| Dataset (including any stated split) | RAVEL |
| Metric | Success Rate (S%) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 14 of 14 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Success Rate (S%)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Success%20Rate%20(S%25)>): Higher is better; unit: PERCENT.

- [Trajectory Efficiency (ηtraj %)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Trajectory%20Efficiency%20(%CE%B7traj%20%25)>): Higher is better; unit: PERCENT.

- [Reference Adherence (ρref%)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Reference%20Adherence%20(%CF%81ref%25)>): Higher is better; unit: PERCENT.

- [Gain (δgain)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Gain%20(%CE%B4gain)>): Higher is better; unit: SCALAR.

- [Judge Score](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Judge%20Score>): Higher is better; unit: SCALAR.

- [Error Rate (ERRπθ%)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Error%20Rate%20(ERR%CF%80%CE%B8%25)>): Lower is better; unit: PERCENT.

- [Number of Observations (\|O\|)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Number%20of%20Observations%20(%7CO%7C)>): Higher is better; unit: SCALAR.

- [Number of Trajectories (\|T\|)](<https://www.sota2.com/research/sota/agentic-text-synthesis-on-ravel.md?metric=Number%20of%20Trajectories%20(%7CT%7C)>): Higher is better; unit: SCALAR.

## Recorded results

| Method | Success Rate (S%) | Result date | Report paper |
| --- | --- | --- | --- |
| DeepSeek-v3.2 | 95.7 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Gemini-3 Pro | 95.1 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Qwen3-8B | 94.7 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Hermes-3-Llama-70B | 91.8 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Qwen3-235B-A22B | 89.4 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| GLM-4.7 | 73.3 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Grok-4 | 69 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Claude-4.5 Sonnet | 68.1 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| GPT-5.2-2025-11-20 | 64.3 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Qwen3-32B | 58.5 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Kimi-K2-Thinking | 58.3 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Qwen3-Max | 47.8 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Llama-3.1-405B | 47.7 | Not recorded | 69a66bce2d603e6de3e3cc0a |
| Llama-3.1-8B | 0 | Not recorded | 69a66bce2d603e6de3e3cc0a |

### DeepSeek-v3.2

Success Rate (S%): 95.7. Result ID: 69a8076e6dbb7075b94d7863.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/692fc410d6b0dd33429332f4.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Gemini-3 Pro

Success Rate (S%): 95.1. Result ID: 69a8076e6dbb7075b94d785d.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Qwen3-8B

Success Rate (S%): 94.7. Result ID: 69a8076e6dbb7075b94d7869.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Model Category":"Open Source LLMs"}. This record does not assert independent reproduction.

### Hermes-3-Llama-70B

Success Rate (S%): 91.8. Result ID: 69a8076e6dbb7075b94d7865.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Model Category":"Open Source LLMs"}. This record does not assert independent reproduction.

### Qwen3-235B-A22B

Success Rate (S%): 89.4. Result ID: 69a8076e6dbb7075b94d7867.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Model Category":"Open Source LLMs"}. This record does not assert independent reproduction.

### GLM-4.7

Success Rate (S%): 73.3. Result ID: 69a8076e6dbb7075b94d7860.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Grok-4

Success Rate (S%): 69. Result ID: 69a8076e6dbb7075b94d785f.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Claude-4.5 Sonnet

Success Rate (S%): 68.1. Result ID: 69a8076e6dbb7075b94d785e.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### GPT-5.2-2025-11-20

Success Rate (S%): 64.3. Result ID: 69a8076e6dbb7075b94d785c.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/695f39f053fdd5c361743ecb.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Qwen3-32B

Success Rate (S%): 58.5. Result ID: 69a8076e6dbb7075b94d7868.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Model Category":"Open Source LLMs"}. This record does not assert independent reproduction.

### Kimi-K2-Thinking

Success Rate (S%): 58.3. Result ID: 69a8076e6dbb7075b94d7862.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a4a1e209131cea823a1.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Qwen3-Max

Success Rate (S%): 47.8. Result ID: 69a8076e6dbb7075b94d7861.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa788e1e209131cea79116.md>)

Recorded evaluation configuration: {"Model Category":"Proprietary LLMs"}. This record does not assert independent reproduction.

### Llama-3.1-405B

Success Rate (S%): 47.7. Result ID: 69a8076e6dbb7075b94d7864.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Model Category":"Open Source LLMs"}. This record does not assert independent reproduction.

### Llama-3.1-8B

Success Rate (S%): 0. Result ID: 69a8076e6dbb7075b94d7866.

[Reporting paper](<https://www.sota2.com/research/paper/69a66bce2d603e6de3e3cc0a.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Model Category":"Open Source LLMs"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
