Factored Classifier-Free Guidance
About
Counterfactual generation aims to simulate realistic hypothetical outcomes under causal interventions. Diffusion models have emerged as a powerful tool for this task, combining DDIM inversion with conditional generation and classifier-free guidance (CFG). In this work, we identify a key limitation of CFG for counterfactual generation: it prescribes a global guidance scale for all attributes, leading to significant spurious changes in inferred counterfactuals. To mitigate this, we propose Factored Classifier-Free Guidance (FCFG), a flexible and model-agnostic guidance technique that enables attribute-wise control following a causal graph. FCFG complements recent advances in classifier-free guidance and can be seamlessly extended to advanced guidance schemes such as CFG++ and APG. Our experiments demonstrate that FCFG significantly improves the axiomatic soundness of inferred counterfactuals across both natural and medical image datasets, mitigating spurious amplification effects, and enhancing counterfactual reversibility.
Key insights
What problem does FCFG solve?
Standard classifier-free guidance uses one global scale for all conditioning attributes, spuriously amplifying attributes that should stay unchanged during a causal intervention. FCFG assigns separate guidance weights to attribute groups, preserving invariant factors.
How does FCFG differ from prior work?
Unlike HVAE, HVAE-soft, and SA-DCG, FCFG modifies only inference-time guidance, needing no extra training objectives. On MIMIC-CXR under do(finding), FCFG achieves a stronger target AUC change with substantially lower off-target amplification than global CFG.
What are the practical uses of these findings?
FCFG improves counterfactual image soundness on natural and medical imaging benchmarks (CelebA-HQ, EMBED, MIMIC-CXR), useful for causal reasoning and fairness research. The paper documents benchmark prototypes only, not production deployment.
What limitations and future work remain?
FCFG requires structured conditioning variables and a pre-specified causal or semantic grouping, and guidance weights are chosen empirically. The authors note future work on adaptively selecting weights and extending to uncertain or learned groupings.