Continual Learning Through Synaptic Intelligence

About

While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.

Friedemann Zenke, Ben Poole, Surya Ganguli · 2017

Related benchmarks

Task	Dataset	Result
Recommendation	Gowalla	Recall@20—	153
Continual Learning	Sequential MNIST	Avg Acc96	149
Continual Learning	CIFAR100 Split	Average Per-Task Accuracy50.4	117
Exemplar-Free Class-Incremental Learning	CIFAR-100	Avg Top-1 Inc Acc37.68	112
Class-incremental learning	CIFAR-100 10 (test)	Average Top-1 Accuracy—	105
Exemplar-Free Class-Incremental Learning	TinyImageNet	Top-1 Acc (Inc)27.02	98
Image Classification	permuted MNIST (pMNIST) (test)	Accuracy95.33	69
Continual Learning	CIFAR100 (test)	Mean Accuracy64.81	69
Class-incremental learning	CIFAR100 10 Tasks	Accuracy15.37	66
Continual Learning	Split CIFAR-100 20 tasks	Mean Test Accuracy45.49	62