Structural Causal Reasoning on Structural Understanding 1,000 samples (adversarial)
70.8AccuracyTransitivity V1
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Transitivity V1Interpretation=Collapsed (always ”Yes”)2026.05 | 70.8 | 82.9 | 70.8 | 100 | 1,000 | |
| Trans Sem V4Interpretation=Structural understanding2026.05 | 69.8 | 79.6 | 76.4 | 83.1 | 770 | |
| D-sep Sem V2Interpretation=Structural understanding2026.05 | 67.8 | 77.5 | 76.7 | 78.2 | 722 | |
| Standard GemmaInterpretation=Task-specific heuristics2026.05 | 66.7 | 76.2 | 77.1 | 75.3 | 691 | |
| D-separation V1Interpretation=Collapsed (catastrophic)2026.05 | 43 | 46.4 | 69.4 | 34.9 | 356 |