Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification

About

Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.

Tzu-Ming Harry Hsu, Hang Qi, Matthew Brown · 2019

Related benchmarks

Task	Dataset	Result
Image Classification	CIFAR-10 (test)	Accuracy85.44	3,381
Image Classification	CIFAR100 (test)	Accuracy53.69	112
Image Classification	CIFAR10 centralized performance (test)	Accuracy84	104
Image Classification	CIFAR-10-LT (IF 50)	Top-1 Accuracy57.11	88
Image Classification	CIFAR-100 LT (IF=50)	Top-1 Acc35.33	82
Image Classification	CIFAR-10-LT IF 100	Top-1 Accuracy52.03	78
Inference Attack	Federated Learning environments Unauthorized Access (test)	Inference Attack Accuracy7.69	66
Inference Attack	Federated Learning environments Authorized Access (test)	Inference Attack Accuracy28.57	66
Image Classification	Tiny-ImageNet	Accuracy0.6977	63
Image Classification	MNIST NN (test)	Communication Rounds19	62