Disentangled Information Bottleneck

About

The information bottleneck (IB) method is a technique for extracting information that is relevant for predicting the target random variable from the source random variable, which is typically implemented by optimizing the IB Lagrangian that balances the compression and prediction terms. However, the IB Lagrangian is hard to optimize, and multiple trials for tuning values of Lagrangian multiplier are required. Moreover, we show that the prediction performance strictly decreases as the compression gets stronger during optimizing the IB Lagrangian. In this paper, we implement the IB method from the perspective of supervised disentangling. Specifically, we introduce Disentangled Information Bottleneck (DisenIB) that is consistent on compressing source maximally without target prediction performance loss (maximum compression). Theoretical and experimental results demonstrate that our method is consistent on maximum compression, and performs well in terms of generalization, robustness to adversarial attack, out-of-distribution detection, and supervised disentangling.

Ziqi Pan, Li Niu, Jianfu Zhang, Liqing Zhang · 2020

Related benchmarks

Task	Dataset	Result
Image Classification	MNIST (test)	Test Accuracy98.2	189
Image Classification	CINIC-10 (test)	Accuracy—	177
Image Classification	MNIST (train)	Train Accuracy94.3	141
Image Classification	CIFAR-10-C (test)	Accuracy (Clean)—	61
Image Classification	CIFAR-10.1 (test)	Test Error13.2	13
Image Classification	MNIST-C	Brightness Error88.7	12
Image Classification	MNIST (test)	Clean Error Rate54	12
Out-of-Distribution Detection	MNIST Out-of-Distribution (test)	AUROC99.4	10
Image Classification	CIFAR-10.2 (test)	Error Rate17.2	10
Out-of-Distribution Detection	MNIST Gaussian	AUROC99.7	8