# Symbolic Reasoning on AQUA Source: [Symbolic Reasoning on AQUA]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:27:04.862Z Evaluation of model performance on symbolic reasoning problems using the AQUA dataset, measured by accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Symbolic Reasoning | | Dataset (including any stated split) | AQUA | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 26 of 26 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | COT | 80.3 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 79.1 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 78.4 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 78.3 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 72.6 | Not recorded | 68fa7b241e209131cea86cf7 | | COMAT | 72.4 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 72.2 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 69.6 | Not recorded | 68fa7b241e209131cea86cf7 | | Llama-3-8B | 67.3 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 67.2 | Not recorded | 68fa7b241e209131cea86cf7 | | Llama-3-8B | 65.2 | Not recorded | 68fa7b241e209131cea86cf7 | | Qwen2-7B | 63.7 | Not recorded | 68fa7b241e209131cea86cf7 | | FLARE | 62.9 | Not recorded | 68fa7b241e209131cea86cf7 | | Qwen2-7B | 62.9 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 62.2 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 59.9 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 58.7 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 57.6 | Not recorded | 68fa7b241e209131cea86cf7 | | Qwen2-1.5B | 57.2 | Not recorded | 68fa7b241e209131cea86cf7 | | Qwen2-1.5B | 56.8 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 55.4 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 52 | Not recorded | 68fa7b241e209131cea86cf7 | | QuaSAR | 51.6 | Not recorded | 68fa7b241e209131cea86cf7 | | COT | 50.7 | Not recorded | 68fa7b241e209131cea86cf7 | | Llama-3-1B | 40.3 | Not recorded | 68fa7b241e209131cea86cf7 | | Llama-3-1B | 39.2 | Not recorded | 68fa7b241e209131cea86cf7 | ### COT Accuracy: 80.3. Result ID: 698b82767a967eddb76f2c49. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### COT Accuracy: 79.1. Result ID: 698b82767a967eddb76f2c48. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Protocol":"ICL"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 78.4. Result ID: 698b82767a967eddb76f2c3f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 78.3. Result ID: 698b82767a967eddb76f2c4c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 72.6. Result ID: 698b82767a967eddb76f2c4b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Protocol":"ICL"}. This record does not assert independent reproduction. ### COMAT Accuracy: 72.4. Result ID: 698b82767a967eddb76f2c4a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Protocol":"ICL"}. This record does not assert independent reproduction. ### COT Accuracy: 72.2. Result ID: 698b82767a967eddb76f2c3c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### COT Accuracy: 69.6. Result ID: 698b82767a967eddb76f2c3b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Protocol":"ICL"}. This record does not assert independent reproduction. ### Llama-3-8B Accuracy: 67.3. Result ID: 698b82767a967eddb76f2c3a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 67.2. Result ID: 698b82767a967eddb76f2c3e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Protocol":"ICL"}. This record does not assert independent reproduction. ### Llama-3-8B Accuracy: 65.2. Result ID: 698b82767a967eddb76f2c39. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"ICL"}. This record does not assert independent reproduction. ### Qwen2-7B Accuracy: 63.7. Result ID: 698b82767a967eddb76f2c47. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### FLARE Accuracy: 62.9. Result ID: 698b82767a967eddb76f2c3d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-8B","Protocol":"ICL"}. This record does not assert independent reproduction. ### Qwen2-7B Accuracy: 62.9. Result ID: 698b82767a967eddb76f2c46. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"ICL"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 62.2. Result ID: 698b82767a967eddb76f2c52. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-1.5B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### COT Accuracy: 59.9. Result ID: 698b82767a967eddb76f2c50. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen2-1.5B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### COT Accuracy: 58.7. Result ID: 698b82767a967eddb76f2c4f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen2-1.5B","Protocol":"ICL"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 57.6. Result ID: 698b82767a967eddb76f2c51. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-1.5B","Protocol":"ICL"}. This record does not assert independent reproduction. ### Qwen2-1.5B Accuracy: 57.2. Result ID: 698b82767a967eddb76f2c4e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### Qwen2-1.5B Accuracy: 56.8. Result ID: 698b82767a967eddb76f2c4d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"ICL"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 55.4. Result ID: 698b82767a967eddb76f2c45. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-1B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### COT Accuracy: 52. Result ID: 698b82767a967eddb76f2c43. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Llama-3-1B","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### QuaSAR Accuracy: 51.6. Result ID: 698b82767a967eddb76f2c44. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3-1B","Protocol":"ICL"}. This record does not assert independent reproduction. ### COT Accuracy: 50.7. Result ID: 698b82767a967eddb76f2c42. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Llama-3-1B","Protocol":"ICL"}. This record does not assert independent reproduction. ### Llama-3-1B Accuracy: 40.3. Result ID: 698b82767a967eddb76f2c41. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"Annotation Tuning"}. This record does not assert independent reproduction. ### Llama-3-1B Accuracy: 39.2. Result ID: 698b82767a967eddb76f2c40. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Strategy":"Base","Protocol":"ICL"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.