# Reasoning and Shortcut Detection on Three synthetic benchmarks Average (test) Source: [Reasoning and Shortcut Detection on Three synthetic benchmarks Average (test)]() Collection index: [llms.txt]() Data updated: 2026-03-24T13:28:09.285Z Evaluation of model performance on Reasoning and Shortcut Detection tasks using the test split average across three specific synthetic benchmarks. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Reasoning and Shortcut Detection | | Dataset (including any stated split) | Three synthetic benchmarks Average (test) | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 12 of 12 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [Robustness Score](): Higher is better; unit: PERCENT. - [Reasoning Score](): Higher is better; unit: PERCENT. - [SC Det. F1](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | SART | 92.5 | Not recorded | 69c21a710636e5bf4afc4727 | | Influence Filtering | 76 | Not recorded | 69c21a710636e5bf4afc4727 | | Data Filtering | 74.7 | Not recorded | 69c21a710636e5bf4afc4727 | | Group DRO | 73.5 | Not recorded | 69c21a710636e5bf4afc4727 | | IRM | 73.3 | Not recorded | 69c21a710636e5bf4afc4727 | | V-REx | 72.1 | Not recorded | 69c21a710636e5bf4afc4727 | | Self-Consistency | 69.1 | Not recorded | 69c21a710636e5bf4afc4727 | | LfF | 69 | Not recorded | 69c21a710636e5bf4afc4727 | | Standard Fine-Tuning (SFT) | 68.8 | Not recorded | 69c21a710636e5bf4afc4727 | | Focal Loss | 68.2 | Not recorded | 69c21a710636e5bf4afc4727 | | Fishr | 68.1 | Not recorded | 69c21a710636e5bf4afc4727 | | JTT | 67.9 | Not recorded | 69c21a710636e5bf4afc4727 | ### SART Accuracy: 92.5. Result ID: 69c2915d47f34ce924aadb20. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Influence Filtering Accuracy: 76. Result ID: 69c2915d47f34ce924aadb1f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Data Filtering Accuracy: 74.7. Result ID: 69c2915d47f34ce924aadb17. [Reporting paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Group DRO Accuracy: 73.5. Result ID: 69c2915d47f34ce924aadb1a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Requires group annotations":"true"}. This record does not assert independent reproduction. ### IRM Accuracy: 73.3. Result ID: 69c2915d47f34ce924aadb1b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Requires multiple environments":"true"}. This record does not assert independent reproduction. ### V-REx Accuracy: 72.1. Result ID: 69c2915d47f34ce924aadb1c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Requires multiple environments":"true"}. This record does not assert independent reproduction. ### Self-Consistency Accuracy: 69.1. Result ID: 69c2915d47f34ce924aadb16. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### LfF Accuracy: 69. Result ID: 69c2915d47f34ce924aadb1e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Standard Fine-Tuning (SFT) Accuracy: 68.8. Result ID: 69c2915d47f34ce924aadb15. [Reporting paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Focal Loss Accuracy: 68.2. Result ID: 69c2915d47f34ce924aadb19. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Fishr Accuracy: 68.1. Result ID: 69c2915d47f34ce924aadb1d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Requires multiple environments":"true"}. This record does not assert independent reproduction. ### JTT Accuracy: 67.9. Result ID: 69c2915d47f34ce924aadb18. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.