Reasoning on ProofWriter
65AccuracyCoT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CoTBackbone=Gemma2-9B2024.05 | 65 | 56.6 | 52.9 | |
| QUIREBackbone=Gemma2-9B2024.05 | 65 | 60.7 | 56.3 | |
| QUIREBackbone=Llama3.1-8B2024.05 | 63 | 66.6 | 58 | |
| QUIRE (w/o IG Vote)Backbone=Llama3.1-8B2024.05 | 62.8 | 64.1 | 56.6 | |
| SCBackbone=Llama3.1-8B2024.05 | 60.6 | 65 | 57.8 | |
| QUIRE (w/o AAE Recall)Backbone=Llama3.1-8B2024.05 | 60.2 | 65.1 | 57 | |
| CoTBackbone=Llama3.1-8B2024.05 | 59.2 | 64.9 | 55.7 | |
| QUIRE (w/o IG Vote)Backbone=Gemma2-9B2024.05 | 58.5 | 57.8 | 52.8 | |
| LtMBackbone=Gemma2-9B2024.05 | 55 | 55.4 | 51.8 | |
| LtMBackbone=Llama3.1-8B2024.05 | 54 | 60.4 | 56.4 | |
| SRBackbone=Llama3.1-8B2024.05 | 51.6 | 65.9 | 53.4 | |
| SCBackbone=Gemma2-9B2024.05 | 31 | 54 | 50.3 | |
| QUIRE (w/o AAE Recall)Backbone=Gemma2-9B2024.05 | 27.5 | 54.2 | 50.3 | |
| SRBackbone=Gemma2-9B2024.05 | 18.5 | 43.1 | 58.6 |