# Full proof generation on NATURALPROOFS-GEN core (test)

Source: [Full proof generation on NATURALPROOFS-GEN core (test)](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:01:10.344Z

Evaluation of model performance on generating full proofs using the test split of the NATURALPROOFS-GEN core dataset, measured by proof correctness and error analysis.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Full proof generation |
| Dataset (including any stated split) | NATURALPROOFS-GEN core (test) |
| Metric | Reasoning Errors (Ref.) |
| Unit | PERCENT |
| Direction | Lower is better |
| Results loaded | 4 of 4 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Reasoning Errors (Ref.)](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Reasoning%20Errors%20(Ref.)>): Lower is better; unit: PERCENT.

- [Reasoning Errors (Eqn.)](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Reasoning%20Errors%20(Eqn.)>): Lower is better; unit: PERCENT.

- [Reasoning Errors (Other)](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Reasoning%20Errors%20(Other)>): Lower is better; unit: PERCENT.

- [Lexical Errors (Lang.)](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Lexical%20Errors%20(Lang.)>): Lower is better; unit: PERCENT.

- [Lexical Errors (Sym.)](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Lexical%20Errors%20(Sym.)>): Lower is better; unit: PERCENT.

- [Per-Step Useful Rate](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Per-Step%20Useful%20Rate>): Higher is better; unit: PERCENT.

- [Per-Step Correct Rate](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Per-Step%20Correct%20Rate>): Higher is better; unit: PERCENT.

- [Full Proof Useful Rate](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Full%20Proof%20Useful%20Rate>): Higher is better; unit: PERCENT.

- [Full Proof Correct Rate](<https://www.sota2.com/research/sota/full-proof-generation-on-naturalproofs-gen-core-test.md?metric=Full%20Proof%20Correct%20Rate>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Reasoning Errors (Ref.) | Result date | Report paper |
| --- | --- | --- | --- |
| NATURALPROVER\_RETRIEVE | 23.52 | Not recorded | 68fa6dc41e209131cea3bf94 |
| NATURALPROVER++ | 23.61 | Not recorded | 68fa6dc41e209131cea3bf94 |
| NATURALPROVER | 25.84 | Not recorded | 68fa6dc41e209131cea3bf94 |
| GPT-3 | 30.92 | Not recorded | 68fa6dc41e209131cea3bf94 |

### NATURALPROVER\_RETRIEVE

Reasoning Errors (Ref.): 23.52. Result ID: 698fdb093a6c314849fc5b09.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Implementation](<https://github.com/wellecks/naturalprover>)

Recorded evaluation configuration: {"Knowledge source":"Retrieved (top-20)","Decoding strategy":"Greedy","Backbone":"GPT-3 (Curie)"}. This record does not assert independent reproduction.

### NATURALPROVER++

Reasoning Errors (Ref.): 23.61. Result ID: 698fdb093a6c314849fc5b0b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Implementation](<https://github.com/wellecks/naturalprover>)

Recorded evaluation configuration: {"Knowledge source":"Human-provided","Decoding strategy":"Stepwise++","Backbone":"GPT-3 (Curie)"}. This record does not assert independent reproduction.

### NATURALPROVER

Reasoning Errors (Ref.): 25.84. Result ID: 698fdb093a6c314849fc5b0a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Implementation](<https://github.com/wellecks/naturalprover>)

Recorded evaluation configuration: {"Knowledge source":"Human-provided","Decoding strategy":"Greedy","Backbone":"GPT-3 (Curie)"}. This record does not assert independent reproduction.

### GPT-3

Reasoning Errors (Ref.): 30.92. Result ID: 698fdb093a6c314849fc5b08.

[Reporting paper](<https://www.sota2.com/research/paper/68fa6dc41e209131cea3bf94.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa680c1e209131cea19d53.md>)

[Implementation](<https://github.com/openai/gpt-3/blob/master/overlap_frequency.md>)

Recorded evaluation configuration: {"Knowledge source":"None","Decoding strategy":"Greedy","Backbone":"GPT-3 (Curie)"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.