# Causal Reasoning on CRASS OOD (held-out) Source: [Causal Reasoning on CRASS OOD (held-out)]() Collection index: [llms.txt]() Data updated: 2026-06-16T08:44:06.594Z Evaluation of model performance on the Causal Reasoning task using the held-out set of the CRASS OOD dataset. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Causal Reasoning | | Dataset (including any stated split) | CRASS OOD (held-out) | | Metric | Performance P0 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Performance P0](): Higher is better; unit: PERCENT. - [Performance P1](): Higher is better; unit: PERCENT. - [Causal Gap](): Higher is better; unit: SCALAR. - [Relative Gap (%)](): Lower is better; unit: PERCENT. ## Recorded results | Method | Performance P0 | Result date | Report paper | | --- | --- | --- | --- | | Qwen2.5-32B | 95 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 95 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Qwen2.5-14B | 91.8 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 91.8 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Qwen2.5-32B | 91 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 91 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Qwen2.5-14B | 89.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 89.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Llama-3.1-8B | 87.2 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 87.2 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Qwen2.5-7B | 87 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 87 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Llama-3.1-8B | 85 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 85 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Phi-3-mini (3.8B) | 84.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Qwen2.5-7B | 84.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 84.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 84.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Phi-3-mini (3.8B) | 84 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 84 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Mistral-7B | 79.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 79.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Base (CLadder adapter) | 79.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Mistral-7B | 73.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 73.5 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Mistral-7B | 71 | Not recorded | 6a30d89fd33dfb23e783cc86 | | Vernier | 71 | Not recorded | 6a30d89fd33dfb23e783cc86 | ### Qwen2.5-32B Performance P0: 95. Result ID: 6a310cc3fbc373592c2d67e8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier","QLoRA":"true"}. This record does not assert independent reproduction. ### Vernier Performance P0: 95. Result ID: 6a310cc3fbc373592c2d6802. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-32B","Quantization":"QLoRA"}. This record does not assert independent reproduction. ### Qwen2.5-14B Performance P0: 91.8. Result ID: 6a310cc3fbc373592c2d67e6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier","QLoRA":"true"}. This record does not assert independent reproduction. ### Vernier Performance P0: 91.8. Result ID: 6a310cc3fbc373592c2d6800. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B"}. This record does not assert independent reproduction. ### Qwen2.5-32B Performance P0: 91. Result ID: 6a310cc3fbc373592c2d67e7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Base","QLoRA":"true"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 91. Result ID: 6a310cc3fbc373592c2d6801. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-32B","Quantization":"QLoRA"}. This record does not assert independent reproduction. ### Qwen2.5-14B Performance P0: 89.5. Result ID: 6a310cc3fbc373592c2d67e5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Base","QLoRA":"true"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 89.5. Result ID: 6a310cc3fbc373592c2d67ff. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-14B"}. This record does not assert independent reproduction. ### Llama-3.1-8B Performance P0: 87.2. Result ID: 6a310cc3fbc373592c2d67ea. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier"}. This record does not assert independent reproduction. ### Vernier Performance P0: 87.2. Result ID: 6a310cc3fbc373592c2d6804. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B"}. This record does not assert independent reproduction. ### Qwen2.5-7B Performance P0: 87. Result ID: 6a310cc3fbc373592c2d67e4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier"}. This record does not assert independent reproduction. ### Vernier Performance P0: 87. Result ID: 6a310cc3fbc373592c2d67fe. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-7B"}. This record does not assert independent reproduction. ### Llama-3.1-8B Performance P0: 85. Result ID: 6a310cc3fbc373592c2d67e9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Base"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 85. Result ID: 6a310cc3fbc373592c2d6803. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B"}. This record does not assert independent reproduction. ### Phi-3-mini (3.8B) Performance P0: 84.5. Result ID: 6a310cc3fbc373592c2d67e1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Base"}. This record does not assert independent reproduction. ### Qwen2.5-7B Performance P0: 84.5. Result ID: 6a310cc3fbc373592c2d67e3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Base"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 84.5. Result ID: 6a310cc3fbc373592c2d67fb. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Phi-3-mini (3.8B)"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 84.5. Result ID: 6a310cc3fbc373592c2d67fd. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5-7B"}. This record does not assert independent reproduction. ### Phi-3-mini (3.8B) Performance P0: 84. Result ID: 6a310cc3fbc373592c2d67e2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier"}. This record does not assert independent reproduction. ### Vernier Performance P0: 84. Result ID: 6a310cc3fbc373592c2d67fc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Phi-3-mini (3.8B)"}. This record does not assert independent reproduction. ### Mistral-7B Performance P0: 79.5. Result ID: 6a310cc3fbc373592c2d67eb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Base"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 79.5. Result ID: 6a310cc3fbc373592c2d6805. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B","Recipe":"Default","lr":"1e-4"}. This record does not assert independent reproduction. ### Base (CLadder adapter) Performance P0: 79.5. Result ID: 6a310cc3fbc373592c2d6807. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B","Recipe":"Rescued","lr":"5e-6","step":"150"}. This record does not assert independent reproduction. ### Mistral-7B Performance P0: 73.5. Result ID: 6a310cc3fbc373592c2d67ed. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier","recipe":"rescued","lr":"5e-6","early stopping step":"150"}. This record does not assert independent reproduction. ### Vernier Performance P0: 73.5. Result ID: 6a310cc3fbc373592c2d6808. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B","Recipe":"Rescued","lr":"5e-6","step":"150"}. This record does not assert independent reproduction. ### Mistral-7B Performance P0: 71. Result ID: 6a310cc3fbc373592c2d67ec. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Adapter type":"Vernier","recipe":"default","lr":"1e-4"}. This record does not assert independent reproduction. ### Vernier Performance P0: 71. Result ID: 6a310cc3fbc373592c2d6806. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B","Recipe":"Default","lr":"1e-4"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.