Causal Reasoning on CRASS OOD (held-out)
95Performance P0Qwen2.5-32B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen2.5-32BAdapter type=Vernier, QLoRA=true2026.06 | 95 | 72.5 | 22.5 | -18 | |
| VernierBackbone=Qwen2.5-32B, Quantization=QLoRA2026.06 | 95 | 72.5 | 22.5 | -18 | |
| Qwen2.5-14BAdapter type=Vernier, QLoRA=true2026.06 | 91.8 | 72.2 | 19.7 | -27 | |
| VernierBackbone=Qwen2.5-14B2026.06 | 91.8 | 72.2 | 19.7 | -27 | |
| Qwen2.5-32BAdapter type=Base, QLoRA=true2026.06 | 91 | 63.5 | 27.5 | — | |
| Base (CLadder adapter)Backbone=Qwen2.5-32B, Quantization=QLoRA2026.06 | 91 | 63.5 | 27.5 | — | |
| Qwen2.5-14BAdapter type=Base, QLoRA=true2026.06 | 89.5 | 62.5 | 27 | — | |
| Base (CLadder adapter)Backbone=Qwen2.5-14B2026.06 | 89.5 | 62.5 | 27 | — | |
| Llama-3.1-8BAdapter type=Vernier2026.06 | 87.2 | 55.8 | 31.3 | -23 | |
| VernierBackbone=Llama-3.1-8B2026.06 | 87.2 | 55.8 | 31.3 | -23 | |
| Qwen2.5-7BAdapter type=Vernier2026.06 | 87 | 62.2 | 24.8 | -27 | |
| VernierBackbone=Qwen2.5-7B2026.06 | 87 | 62.2 | 24.8 | -27 | |
| Llama-3.1-8BAdapter type=Base2026.06 | 85 | 44.5 | 40.5 | — | |
| Base (CLadder adapter)Backbone=Llama-3.1-8B2026.06 | 85 | 44.5 | 40.5 | — | |
| Phi-3-mini (3.8B)Adapter type=Base2026.06 | 84.5 | 54.5 | 30 | — | |
| Qwen2.5-7BAdapter type=Base2026.06 | 84.5 | 50.5 | 34 | — | |
| Base (CLadder adapter)Backbone=Phi-3-mini (3.8B)2026.06 | 84.5 | 54.5 | 30 | — | |
| Base (CLadder adapter)Backbone=Qwen2.5-7B2026.06 | 84.5 | 50.5 | 34 | — | |
| Phi-3-mini (3.8B)Adapter type=Vernier2026.06 | 84 | 55 | 29 | -3 | |
| VernierBackbone=Phi-3-mini (3.8B)2026.06 | 84 | 55 | 29 | -3 | |
| Mistral-7BAdapter type=Base2026.06 | 79.5 | 41.5 | 38 | — | |
| Base (CLadder adapter)Backbone=Mistral-7B, Recipe=Default, lr=1e-42026.06 | 79.5 | 41.5 | 38 | — | |
| Base (CLadder adapter)Backbone=Mistral-7B, Recipe=Rescued, lr=5e-6, step=1502026.06 | 79.5 | 41.5 | 38 | — | |
| Mistral-7BAdapter type=Vernier, recipe=rescued, lr=5e-6, early stopping step=1502026.06 | 73.5 | 36.5 | 37 | -3 | |
| VernierBackbone=Mistral-7B, Recipe=Rescued, lr=5e-6, step=1502026.06 | 73.5 | 36.5 | 37 | -3 | |
| Mistral-7BAdapter type=Vernier, recipe=default, lr=1e-42026.06 | 71 | 50 | 21 | -45 | |
| VernierBackbone=Mistral-7B, Recipe=Default, lr=1e-42026.06 | 71 | 50 | 21 | -45 |