Continual Supervised Learning on CIFAR 5+1
74.9Total Average Online Task AccuracyPrune
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| PruneCompactness=50%2026.05 | 74.9 | 24.1 | |
| PruneCompactness=40%2026.05 | 74.8 | 24.4 | |
| PruneCompactness=30%2026.05 | 74 | 24.1 | |
| PruneCompactness=20%2026.05 | 73.1 | 24.1 | |
| Deep FourierActivation=Deep Fourier2025.09 | 72.29 | — | |
| Grow+Rand. Smooth-LeakyCompactness=50%2026.05 | 67.9 | 22.6 | |
| Grow+Rand. Smooth-LeakyCompactness=30%2026.05 | 65.3 | 22.1 | |
| Grow+Rand. Smooth-LeakyCompactness=40%2026.05 | 65 | 22.4 | |
| Grow+Rand. Smooth-LeakyCompactness=20%2026.05 | 63.6 | 21.6 | |
| Rand. Smooth-LeakyActivation=Rand. Smooth-Leaky2025.09 | 57.01 | — | |
| CeLUActivation=CeLU2025.09 | 54.23 | — | |
| RReLUActivation=RReLU2025.09 | 53.6 | — | |
| Smooth-LeakyActivation=Smooth-Leaky2025.09 | 49.87 | — | |
| SeLUActivation=SeLU2025.09 | 49.07 | — | |
| Leaky-ReLUActivation=Leaky-ReLU2025.09 | 48.86 | — | |
| eLUActivation=eLU2025.09 | 47.64 | — | |
| PReLUActivation=PReLU2025.09 | 43.3 | — | |
| RationalActivation=Rational2025.09 | 40.41 | — | |
| Grow+Moment TransplantCompactness=40%2026.05 | 38.3 | 18.2 | |
| Swish (SiLU)Activation=Swish (SiLU)2025.09 | 35.31 | — | |
| Grow+Moment TransplantCompactness=30%2026.05 | 35.1 | 17.5 | |
| GrowCompactness=50%2026.05 | 34.4 | 17.7 | |
| GrowCompactness=40%2026.05 | 33.8 | 18.8 | |
| Grow+GradMaxCompactness=50%2026.05 | 33.7 | 18.4 | |
| Grow+GradMaxCompactness=30%2026.05 | 31.8 | 17.7 | |
| Grow+Net2WiderCompactness=40%2026.05 | 31.8 | 16.9 | |
| Grow+Moment TransplantCompactness=50%2026.05 | 31.6 | 13.3 | |
| Grow+Net2WiderCompactness=50%2026.05 | 30.9 | 17.2 | |
| Grow+Net2WiderCompactness=30%2026.05 | 30.6 | 18 | |
| Grow+TwoSpeedCompactness=40%2026.05 | 30.2 | 17.7 | |
| Grow+GradMaxCompactness=40%2026.05 | 30 | 17 | |
| GrowCompactness=30%2026.05 | 29.4 | 17.7 | |
| Grow+TwoSpeedCompactness=50%2026.05 | 29.3 | 12.5 | |
| TanhActivation=Tanh2025.09 | 28.59 | — | |
| Grow+TwoSpeedCompactness=30%2026.05 | 28.3 | 17.3 | |
| Grow+Moment TransplantCompactness=20%2026.05 | 25 | 18.1 | |
| Grow+GradMaxCompactness=20%2026.05 | 24.4 | 16.4 | |
| GrowCompactness=20%2026.05 | 23.7 | 16.9 | |
| Grow+Net2WiderCompactness=20%2026.05 | 23 | 17.1 | |
| Grow+TwoSpeedCompactness=20%2026.05 | 20.8 | 16.4 | |
| CReLUActivation=CReLU2025.09 | 20.56 | — | |
| GeLUActivation=GeLU2025.09 | 17.6 | — | |
| DenseCompactness=20%2026.05 | 13.8 | 26.1 | |
| DenseCompactness=30%2026.05 | 13.8 | 26.1 | |
| DenseCompactness=40%2026.05 | 13.8 | 26.1 | |
| DenseCompactness=50%2026.05 | 13.8 | 26.1 | |
| SwiGLUActivation=SwiGLU2025.09 | 9.57 | — | |
| ReLUActivation=ReLU2025.09 | 4.76 | — | |
| SigmoidActivation=Sigmoid2025.09 | 1.79 | — |