# Counterfactual Cross-Validation: Stable Model Selection Procedure for Causal Inference Models

Source: [Counterfactual Cross-Validation: Stable Model Selection Procedure for Causal Inference Models](<https://www.sota2.com/research/paper/68fa67f71e209131cea19795>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:25:59.816Z

Authors: Yuta Saito, Shota Yasui

Published: 2019-09-11T18:43:32.000Z

## Abstract

We study the model selection problem in conditional average treatment effect (CATE) prediction. Unlike previous works on this topic, we focus on preserving the rank order of the performance of candidate CATE predictors to enable accurate and stable model selection. To this end, we analyze the model performance ranking problem and formulate guidelines to obtain a better evaluation metric. We then propose a novel metric that can identify the ranking of the performance of CATE predictors with high confidence. Empirical evaluations demonstrate that our metric outperforms existing metrics in both model selection and hyperparameter tuning tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [CATE Estimator Ranking on ACIC Setting A (p=0) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-a-p-0-2016.md>): CATE Estimator Ranking; ACIC Setting A (p=0) 2016; Rank Correlation: 0.54.

- [CATE Estimator Ranking on ACIC Setting A (p=0.1) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-a-p-0-1-2016.md>): CATE Estimator Ranking; ACIC Setting A (p=0.1) 2016; Rank Correlation: 0.53.

- [CATE Estimator Ranking on ACIC Setting A (p=0.3) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-a-p-0-3-2016.md>): CATE Estimator Ranking; ACIC Setting A (p=0.3) 2016; Rank Correlation: 0.53.

- [CATE Estimator Ranking on ACIC Setting B (xi=0) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-b-xi-0-2016.md>): CATE Estimator Ranking; ACIC Setting B (xi=0) 2016; Rank Correlation: 0.87.

- [CATE Estimator Selection on ACIC Setting C, m=0.5 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-c-m-0-5-2016.md>): CATE Estimator Selection; ACIC Setting C (m=0.5) 2016; PEHE: 53.87.

- [CATE Estimator Ranking on ACIC Setting B (xi=2) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-b-xi-2-2016.md>): CATE Estimator Ranking; ACIC Setting B (xi=2) 2016; Rank Correlation: 0.55.

- [CATE Estimator Ranking on ACIC Setting C (m=0.5) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-c-m-0-5-2016.md>): CATE Estimator Ranking; ACIC Setting C (m=0.5) 2016; Rank Correlation: 0.54.

- [CATE Estimator Selection on ACIC Setting A (p=0.1) 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-a-p-0-1-2016.md>): CATE Estimator Selection; ACIC Setting A (p=0.1) 2016; PEHE: 63.8.

- [CATE Estimator Selection on ACIC Setting A, p=0.3 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-a-p-0-3-2016.md>): CATE Estimator Selection; ACIC Setting A (p=0.3) 2016; PEHE: 63.1.

- [CATE Estimator Ranking on ACIC Setting C (m=0.1) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-c-m-0-1-2016.md>): CATE Estimator Ranking; ACIC Setting C (m=0.1) 2016; Rank Correlation: 0.5.

- [CATE Estimator Ranking on ACIC Setting C (m=0.9) 2016](<https://www.sota2.com/research/sota/cate-estimator-ranking-on-acic-setting-c-m-0-9-2016.md>): CATE Estimator Ranking; ACIC Setting C (m=0.9) 2016; Rank Correlation: 0.58.

- [CATE Estimator Selection on ACIC Setting A (p=0) 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-a-p-0-2016.md>): CATE Estimator Selection; ACIC Setting A (p=0) 2016; PEHE: 63.07.

- [CATE Estimator Selection on ACIC Setting B, xi=0 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-b-xi-0-2016.md>): CATE Estimator Selection; ACIC Setting B (xi=0) 2016; PEHE: 16.51.

- [CATE Estimator Selection on ACIC Setting B, xi=2 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-b-xi-2-2016.md>): CATE Estimator Selection; ACIC Setting B (xi=2) 2016; PEHE: 73.29.

- [CATE Estimator Selection on ACIC Setting C, m=0.1 2016](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic-setting-c-m-0-1-2016.md>): CATE Estimator Selection; ACIC Setting C (m=0.1) 2016; PEHE: 65.12.

- [CATE Estimator Selection on ACIC2016 Setting C m=0.9](<https://www.sota2.com/research/sota/cate-estimator-selection-on-acic2016-setting-c-m-0-9.md>): CATE Estimator Selection; ACIC2016 Setting C m=0.9; PEHE: 53.79.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1909.05299>)

[PDF](<https://arxiv.org/pdf/1909.05299.pdf>)
