Targeted error generation on TheoremQA Tier-1 (first-20 sweep)
54Targeted Error RateP1 (answer + EA)
Evaluation Results
| Method | Links | |
|---|---|---|
| P1 (answer + EA)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 54 | |
| P0 (baseline)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 61 | |
| P6 (expanded examples)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 62 | |
| P4 (P3 + textbook (EA))Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 64 | |
| P2 (answer, no EA)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 66 | |
| P2 (answer, no EA)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 68 | |
| P6 (expanded examples)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 68 | |
| P0 (baseline)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 69 | |
| P5 (hybrid + textbook)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 70 | |
| P1 (answer + EA)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 71 | |
| P3 (P1 + feedback)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 71 | |
| P4 (P3 + textbook (EA))Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 72 | |
| P5 (hybrid + textbook)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 72 | |
| P8 (fine-tuned EA)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 74 | |
| P3 (P1 + feedback)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 75 | |
| P7 (P6 + textbook)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 76 | |
| P7 (P6 + textbook)Backbone=GPT-5-mini, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 76 | |
| P0 (baseline)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 77 | |
| P5 (hybrid + textbook)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 77 | |
| P6 (expanded examples)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 77 | |
| P7 (P6 + textbook)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 77 | |
| P8 (fine-tuned EA)Backbone=o3+4o, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 79 | |
| P2 (answer, no EA)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 81 | |
| P8 (fine-tuned EA)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 83 | |
| P4 (P3 + textbook (EA))Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 86 | |
| P1 (answer + EA)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 87 | |
| P3 (P1 + feedback)Backbone=GPT-5, GA/EA Retries=unlimited, Verification=human-verified2026.05 | 89 |