# kDecay: Just adding k-decay items on Learning-Rate Schedule to improve Neural Networks Source: [kDecay: Just adding k-decay items on Learning-Rate Schedule to improve Neural Networks]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:01:55.246Z Authors: Tao Zhang, Wei Li Published: 2020-04-13T12:58:45.000Z ## Abstract Recent work has shown that optimizing the Learning Rate (LR) schedule can be a very accurate and efficient way to train deep neural networks. We observe that the rate of change (ROC) of LR has correlation with the training process, but how to use this relationship to control the training to achieve the purpose of improving accuracy? We propose a new method, k-decay, just add an extra item to the commonly used and easy LR schedule(exp, cosine and polynomial), is effectively improves the performance of these schedule, also better than the state-of-the-art algorithms of LR shcedule such as SGDR, CLR and AutoLRS. In the k-decay, by adjusting the hyper-parameter \\(k\\), to generate different LR schedule, when k increases, the performance is improved. We evaluate the k-decay method on CIFAR And ImageNet datasets with different neural networks (ResNet, Wide ResNet). Our experiments show that this method can improve on most of them. The accuracy has been improved by 1.08\\% on the CIFAR-10 dataset and by 2.07 \\% on the CIFAR-100 dataset. On the ImageNet, accuracy is improved by 1.25\\%. Our method is not only a general method to be applied other LR Shcedule, but also has no additional computational cost. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()