Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness

Keywords

Smoothness, Artificial neural network, Generalization, Computer science, Algorithm

References

• Balls in Rk do not cut all subsets of k + 2 pointsR. M. Dudley — 1979
• Columbia Object Image Library (COIL100)S.K. Nayar — 1996
• Path-SGD: Path-Normalized Optimization in Deep Neural NetworksBehnam Neyshabur, Ruslan Salakhutdinov, Nathan Srebro — 2015
• The Elements of Statistical Learning: Data Mining, Inference, and PredictionTrevor Hastie, J. Friedman, Robert Tibshirani — 2001
• The Elements of Statistical LearningEric R. Ziegel — 2003
• Approximation by superpositions of a sigmoidal functionGeorge Cybenko — 1989
• Gradient-based learning applied to document recognitionYann LeCun, Léon Bottou, Yoshua Bengio — 1998
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep LearningBehnam Neyshabur, Ryota Tomioka, Nathan Srebro — 2014
• Multilayer feedforward networks are universal approximatorsKurt Hornik, Maxwell B. Stinchcombe, Halbert White — 1989
• ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton — 2012
• Mastering the game of Go with deep neural networks and tree searchDavid Silver, Aja Huang, Chris J. Maddison — 2016
• Gradient Descent Converges to MinimizersJason D. Lee, Max Simchowitz, Michael I. Jordan — 2016
• Reading Digits in Natural Images with Unsupervised Feature LearningYuval Netzer — 2011
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaNitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal — 2016
• Robust Large Margin Deep Neural NetworksJure Sokolić, Raja Giryes, Guillermo Sapiro — 2016
• Understanding deep learning requires rethinking generalizationChiyuan Zhang, Samy Bengio, Moritz Hardt — 2016
• Fast Rates for Empirical Risk Minimization of Strict Saddle ProblemsAlon Gonen, Shai Shalev‐Shwartz — 2017
• Opening the Black Box of Deep Neural Networks via InformationRavid Shwartz-Ziv, Naftali Tishby — 2017
• Theory II: Landscape of the Empirical Risk in Deep LearningQianli Liao, Tomaso Poggio — 2017
• Data-Dependent Stability of Stochastic Gradient DescentIlja Kuzborskij, Christoph H. Lampert — 2017

Cited by

• DeepXDE: A Deep Learning Library for Solving Differential EquationsLu Lu, Xuhui Meng, Zhiping Mao — 2019
• Learning nonlinear operators via DeepONet based on the universal approximation theorem of operatorsLu Lu, Pengzhan Jin, Guofei Pang — 2019
• Variational Physics-Informed Neural Networks For Solving Partial Differential EquationsEhsan Kharazmi, Zhongqiang Zhang, George Em Karniadakis — 2019
• Deep learning architectures for nonlinear operator functions and nonlinear inverse problemsMaarten V. de Hoop, Matti Lassas, Christopher A. Wong — 2019
• Symplectic networks: Intrinsic structure-preserving networks for identifying Hamiltonian systemsPengzhan Jin, Aiqing Zhu, George Em Karniadakis — 2020
• Deep Hamiltonian networks based on symplectic integratorsAiqing Zhu, Pengzhan Jin, Yifa Tang — 2020
• A machine-learning-based surrogate model of Mars’ thermal evolutionSiddhant Agarwal, Nicola Tosi, D. Breuer — 2020
• SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systemsPengzhan Jin, Zhen Zhang, Aiqing Zhu — 2020
• Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODEJuntang Zhuang, Nicha C. Dvornek, Xiaoxiao Li — 2020
• Inverse modified differential equations for discovery of dynamicsAiqing Zhu, Pengzhan Jin, Zhu, Beibei — 2020
• Relative Flatness and Generalization in the Interpolation Regime.Henning Petzka, Michael Kamp, Linara Adilova — 2020
• Quantitative analysis of the generalization ability of deep feedforward neural networksYanli Yang, Chenxia Li — 2021
• Physics-informed neural networks with hard constraints for inverse designLu Lu, Raphaël Pestourie, Wenjie Yao — 2021
• Research on Classification of Watermelon Ripeness Based on Neural Network Pattern RecognitionYuchan Chen — 2021
• Approximation capabilities of measure-preserving neural networksAiqing Zhu, Pengzhan Jin, Yifa Tang — 2021
• European Union–Ukraine Association Agreement: Challenges and Prospects for CooperationОлена Дурман, Микола Дурман, Tatyana Mazievich — 2021
• Mosaic flows: A transferable deep learning framework for solving PDEs on unseen domainsHengjie Wang, Robert Planas, Aparna Chandramowlishwaran — 2021
• An Contract Theory based Federated Learning Aggregation Algorithm in IoT NetworkMengmeng Tian — 2022
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networksYaoyu Zhang, Yuqing Li, Zhongwang Zhang — 2021
• The mnemonic basis of subjective experienceHakwan Lau, Matthias Michel, Joseph E. LeDoux — 2022

Related papers

• Neural Tangent Kernel Eigenvalues Accurately Predict GeneralizationJames B. Simon, Madeline Dickens, Michael R. DeWeese — 2021
• Generalization error of deep neural networks: Role of classification margin and data structureJure Sokolić, Raja Giryes, Guillermo Sapiro — 2017
• A Spectral Approach to Generalization and Optimization in Neural NetworksFarzan Farnia, Jesse Zhang, David Tse — 2018
• A Fourier-Based Approach to Generalization and Optimization in Deep LearningFarzan Farnia, Jesse M. Zhang, David Tse — 2020
• Generalization in fully-connected neural networks for time series forecastingAnastasia Borovykh, Cornelis W. Oosterlee, Sander M. Bohté — 2019
• On the modeling of error functions as high dimensional landscapes for weight initialization in learning networksJulius, Gopinath Mahale, T. Sumana — 2016
• Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System ApproachGrant M. Rotskoff, Eric Vanden‐Eijnden — 2018