Learning Activation Functions to Improve Deep Neural Networks

About

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.

Forest Agostinelli, Matthew Hoffman, Peter Sadowski, Pierre Baldi · 2014

Related benchmarks

Task	Dataset	Result
Image Classification	CIFAR-100 (test)	Accuracy—	3,518
Image Classification	CIFAR-10 (test)	Accuracy—	3,381
Image Classification	CIFAR-10 Standard data augmentation (test)	Test Error Rate7.51	43
Classification	Evaluation Benchmark (aggregated)	Accuracy78.61	27
Image Classification	CIFAR-100 Standard data augmentation (test)	Test Error30.83	22
Binary Classification	Higgs boson decay dataset	AUC—	6