Cyclical Learning Rates for Training Neural Networks

Keywords

Artificial neural network, Computer science, Artificial intelligence, Schedule, Monotonic function

References

• ADADELTA: An Adaptive Learning Rate MethodMatthew D. Zeiler — 2012
• Neural Networks: Tricks of the TradeMontavon, Grégoire, Müller, Klaus-Robert, Orr, Geneviève B — 2012
• An Empirical Evaluation of Deep Learning on Highway DrivingBrody Huval, Tao Wang, Sameep Tandon — 2015
• Very Deep Convolutional Networks for Large-Scale Image RecognitionKaren Simonyan, Andrew Zisserman — 2014
• ADASECANT: Robust Adaptive Secant Method for Stochastic GradientÇağlar Gülçehre, Moczulski, Marcin, Bengio, Yoshua — 2014
• The Effects of Hyperparameters on SGD Training of Neural NetworksThomas M. Breuel — 2015
• Show and tell: A neural image caption generatorOriol Vinyals, Alexander Toshev, Samy Bengio — 2014
• Going deeper with convolutionsChristian Szegedy, Wei Liu, Yangqing Jia — 2014
• Towards End-To-End Speech Recognition with Recurrent Neural NetworksAlex Graves, Navdeep Jaitly — 2014
• Rich Feature Hierarchies for Accurate Object Detection and Semantic SegmentationRoss Girshick, Jeff Donahue, Trevor Darrell — 2013
• ImageNet Large Scale Visual Recognition ChallengeOlga Russakovsky, Jia Deng, Hao Su — 2014
• DeepFace: Closing the Gap to Human-Level Performance in Face VerificationYaniv Taigman, Ming Yang, Marc’Aurelio Ranzato — 2014
• Adaptive stepsizes for recursive estimation with applications in approximate dynamic programmingAbraham George, Warren B. Powell — 2006
• Caffe: Convolutional Architecture for Fast Feature EmbeddingYangqing Jia, Evan Shelhamer, Jeff Donahue — 2014
• ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton — 2012
• Deep Residual Learning for Image RecognitionKaiming He, Xiangyu Zhang, Shaoqing Ren — 2015
• Equilibrated adaptive learning rates for non-convex optimizationYann Dauphin, Harm de Vries, Yoshua Bengio — 2015
• SGDR: Stochastic Gradient Descent with Warm RestartsIlya Loshchilov, Frank Hutter — 2016
• A method for solving the convex programming problem with convergence rate O(1/k^2)Yurii Nesterov — 1983
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftSergey Ioffe, Christian Szegedy — 2015

Cited by

• Data Compression Based on Stacked RBM-AE Model for Wireless Sensor NetworksArmando José Torrent Ruiz — 2018
• Intelligent upgrade of waste-activated sludge dewatering process based on artificial neural network model: Core influential factor identification and non-experimental prediction of sludge dewatering performance.Philip H. Coombs, Manzoor Ahmed — 2023
• Attentive Training: A New Training Framework for Speech EnhancementAshutosh Pandey, DeLiang Wang — 2023
• Machine Learning Refined: Foundations, Algorithms, and ApplicationsJeremy Watt, Reza Borhani, Aggelos K. Katsaggelos — 2016
• SGDR: Stochastic Gradient Descent with Warm RestartsIlya Loshchilov, Frank Hutter — 2016
• Exploring loss function topology with cyclical learning ratesLeslie N. Smith, Nicholay Topin — 2017
• Best Practices for Applying Deep Learning to Novel ApplicationsLeslie N. Smith — 2017
• Super-Convergence: Very Fast Training of Residual Networks Using Large Learning RatesLeslie N. Smith, Nicholay Topin — 2017
• Three Factors Influencing Minima in SGDStanisław Jastrzȩbski, Zachary Kenton, Devansh Arpit — 2017
• Fixing Weight Decay Regularization in AdamIlya Loshchilov, Frank Hutter — 2017
• A deep probabilistic framework for heterogeneous self-supervised learning of affordancesAtabak Dehban, Lorenzo Jamone, Adam R. Kampff — 2017
• Fine-tuned Language Models for Text ClassificationJeremy Howard, Sebastian Ruder — 2018
• Understanding Short-Horizon Bias in Stochastic Meta-OptimizationYuhuai Wu, Mengye Ren, Renjie Liao — 2018
• Localization and classification of cell nuclei in post-neoadjuvant breast cancer surgical specimen using fully convolutional networksMehrdad J. Gangeh, Rene Bidart, Mohammad Peikari — 2018
• Exponential decay sine wave learning rate for fast deep neural network trainingWangpeng An, Haoqian Wang, Yulun Zhang — 2017
• A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decayLeslie N. Smith — 2018
• ResNet Sparsifier: Learning Strict Identity Mappings in Deep Residual NetworksXin Yu, Zhiding Yu, Srikumar Ramalingam — 2018
• Universal Language Model Fine-tuning for Text ClassificationJeremy Howard, Sebastian Ruder — 2018
• Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural NetworksJonathan Rubin, Deepan Sanghavi, Claire Zhao — 2018
• Reliability Map Estimation for CNN-Based Camera Model AttributionDavid Güera, Fengqing Zhu, Sri Kalyan Yarlagadda — 2018

Related papers

• Deep Residual Learning for Image RecognitionKaiming He, Xiangyu Zhang, Shaoqing Ren — 2015
• Adam: A Method for Stochastic OptimizationDiederik P. Kingma, Jimmy Ba — 2014
• ImageNet: A large-scale hierarchical image databaseJia Deng, Wei Dong, Richard Socher — 2009
• ImageNet Large Scale Visual Recognition ChallengeOlga Russakovsky, Jia Deng, Hao Su — 2014
• A Cyclical Learning Rate Method in Deep Learning TrainingJiaqi Li, Xiaodong Yang — 2020
• Deep LearningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton — 2015
• Adaptive Subgradient Methods for Online Learning and Stochastic OptimizationJohn C. Duchi, Elad Hazan, Yoram Singer — 2010
• No More Pesky Learning Rate Guessing GamesLeslie N. Smith — 2015
• An overview of gradient descent optimization algorithmsSebastian Ruder — 2016