Regularisation of neural networks by enforcing Lipschitz continuity
Explore this paper's citation graph
Summary
The technique is used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods and shows that the performance of the resulting models exceeds that of models trained with other common regularisers.
- Type
- preprint
- Published
- 2018-04-12
- Cited by
- 639
- References
- 44
- Access
- Open access
- OpenAlex
- https://openalex.org/W2796892552
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:4811672
Keywords
Lipschitz continuity, Constant (computer programming), Artificial neural network, Hyperparameter, Bounded function
References
- Regularization of Neural Networks using DropConnect
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Hyperopt: a Python library for model selection and hyperparameter optimization
- Understanding the difficulty of training deep feedforward neural networks
- Statistical Comparisons of Classifiers over Multiple Data Sets
- Intriguing properties of neural networks
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Real Mathematical Analysis
- Robustness and generalization
- Closed-form dual perturb and combine for tree-based models
- Dropout: a simple way to prevent neural networks from overfitting
- The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network
- Gradient-based learning applied to document recognition
- Convex Optimization: Algorithms and Complexity
- Deep Residual Learning for Image Recognition
- Tikhonov, Ivanov and Morozov regularization for support vector machine learning
- On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
- Spectral Norm Regularization for Improving the Generalizability of Deep Learning
- L2 Regularization versus Batch and Weight Normalization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Cited by
- MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes
- Lipschitz Networks and Distributional Robustness
- Automated Discovery of Jet Substructure Analyses
- Principled Deep Neural Network Training through Linear Programming
- Improved robustness to adversarial examples using Lipschitz regularization of the loss
- Invertible Residual Networks
- Machine Learning of Coarse-Grained Molecular Dynamics Force Fields
- Adversarial Attacks, Regression, and Numerical Stability Regularization
- Lipschitz regularized Deep Neural Networks converge and generalize
- Deep Learning for Inverse Problems: Bounds and Regularizers
- Certified Adversarial Robustness via Randomized Smoothing
- The Neural Network Approach to Inverse Problems in Differential Equations
- Numerically Recovering the Critical Points of a Deep Linear Autoencoder
- Evolving and Understanding Sparse Deep Neural Networks using Cosine Similarity
- DL2: Training and Querying Neural Networks with Logic
- Universal Lipschitz Approximation in Bounded Depth Neural Networks
- Reducing Adversarial Example Transferability Using Gradient Regularization
- Efficient Randomized Defense against Adversarial Attacks in Deep Convolutional Neural Networks
- Kernel Random Matrices of Large Concentrated Data: the Example of GAN-Generated Images
- Minimum Uncertainty Based Detection of Adversaries in Deep Neural Networks