MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes
Explore this paper's citation graph
Summary
An empirical analogue to the Lipschitz constant of a feed-forward neural network, which is referred to as the maximum gain, is presented, hypothesising that constraining the gain of a network will have a regularising effect, similar to how constrainingThe LipsChitz constant has been shown to improve generalisation.
- Type
- preprint
- Published
- 2018-04-16
- Cited by
- 7
- References
- 19
- Access
- Open access
- OpenAlex
- https://openalex.org/W2797798064
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:4895746
Keywords
Overfitting, Benchmark (surveying), Lipschitz continuity, Computer science, Artificial neural network
References
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Dropout: a simple way to prevent neural networks from overfitting
- ImageNet: A large-scale hierarchical image database
- Understanding deep learning requires rethinking generalization
- Sharp Minima Can Generalize For Deep Nets
- Concrete Dropout
- Regularisation of neural networks by enforcing Lipschitz continuity
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Wide Residual Networks
- Generative Adversarial Networks
- Learning Multiple Layers of Features from Tiny Images
- Concrete Dropout
- Wasserstein GAN
- Adam: A Method for Stochastic Optimization
- Spectral Normalization for Generative Adversarial Networks
- Inference for the Generalization Error
- Spectral Normalization for Generative Adversarial Networks
- Improved Training of Wasserstein GANs
- Variational Dropout and the Local Reparameterization Trick
Cited by
- Regularisation of neural networks by enforcing Lipschitz continuity
- The Singular Values of Convolutional Layers
- Generalised Lipschitz Regularisation Equals Distributional Robustness
- iFlowGAN: An Invertible Flow-Based Generative Adversarial Network for Unsupervised Image-to-Image Translation
- Convolutional Dynamic Alignment Networks for Interpretable Classifications
- Nonlinearity Enhanced Adaptive Activation Function
- Neuron with Steady Response Leads to Better Generalization
Related papers
- Bayesian Recurrent Neural Networks
- Estimating Equivalent Kernels for Neural Networks: A Data Perturbation Approach
- Probabilistic robustness estimates for feed-forward neural networks
- On the Use of Dropouts in Neural Networks for System Identification and Control
- A Hybrid Objective Function for Robustness of Artificial Neural Networks - Estimation of Parameters in a Mechanical System
- Local Bayesian Regularisation of Parsimonious Neurofuzzy Models for Real World Dynamic Processes
- Deep Component Analysis via Alternating Direction Neural Networks
- Consistent feature selection for neural networks via Adaptive Group Lasso
- Deep Neural Networks as Gaussian Processes