Causal Discovery with Language Models as Imperfect Experts

About

Understanding the causal relationships that underlie a system is a fundamental prerequisite to accurate decision-making. In this work, we explore how expert knowledge can be used to improve the data-driven identification of causal graphs, beyond Markov equivalence classes. In doing so, we consider a setting where we can query an expert about the orientation of causal relationships between variables, but where the expert may provide erroneous information. We propose strategies for amending such expert knowledge based on consistency properties, e.g., acyclicity and conditional independencies in the equivalence class. We then report a case study, on real data, where a large language model is used as an imperfect expert.

Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, Alexandre Drouin · 2023

Related benchmarks

Task	Dataset	Result
Causal Discovery	Alarm d=37	Mod. SHD9	21
Causal Discovery	Barley n=48	SHD64	14
Causal Discovery	Insurance n=27	SHD24	14
Causal Discovery	Win95pts	SHD64	14
Causal Discovery	Water n=32	SHD55	14
Causal Discovery	Cancer n=5	Structural Hamming Distance (SHD)4	14
Causal Discovery	Asia synthetic averaged over n ∈ {500, 1000, 5000, 10000}	RNB0.625	11
Causal Discovery	Alarm averaged over n ∈ {500, 1000, 5000, 10000} (synthetic)	RNB0.457	11
Causal Discovery	Sachs synthetic averaged over n ∈ {500, 1000, 5000, 10000}	RNB0.412	11
Causal Discovery	Sangiovese real-world fixed sample size	RNB0.471	11