Reconciling modern machine-learning practice and the classical bias–variance trade-off
Explore this paper's citation graph
Summary
This work shows how classical theory and modern practice can be reconciled within a single unified performance curve and proposes a mechanism underlying its emergence, and provides evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets.
- Type
- article
- Published
- 2018-12-28
- Cited by
- 2,155
- References
- 47
- Access
- Open access
- OpenAlex
- https://openalex.org/W2963518130
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:198496504
Keywords
Variance (accounting), Economics, Econometrics, Artificial intelligence, Machine learning
References
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction
- Understanding the difficulty of training deep feedforward neural networks
- All of Nonparametric Statistics
- Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers
- Greedy function approximation: A gradient boosting machine.
- Boosting the margin: A new explanation for the effectiveness of voting methods
- Occam's Razor
- Neural Networks and the Bias/Variance Dilemma
- A training algorithm for optimal margin classifiers
- Boosting With the L2 Loss
- The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network
- Gradient-based learning applied to document recognition
- ImageNet Large Scale Visual Recognition Challenge
- Random Features for Large-Scale Kernel Machines
- Homo Heuristicus: Why Biased Minds Make Better Inferences
- Dynamics of Training
- Kernel Methods for Deep Learning
- GloVe: Global Vectors for Word Representation
- Generalization Properties of Learning with Random Features
- Reading Digits in Natural Images with Unsupervised Feature Learning
Cited by
- High-dimensional dynamics of generalization error in neural networks
- Data augmentation instead of explicit regularization
- Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
- A Modern Take on the Bias-Variance Tradeoff in Neural Networks
- Scaling description of generalization with number of parameters in deep learning
- Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits
- Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
- Two models of double descent for weak features
- SURPRISES IN HIGH-DIMENSIONAL RIDGELESS LEAST SQUARES INTERPOLATION
- The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
- Linearized two-layers neural networks in high dimension
- Lightlike Neuromanifolds, Occam's Razor and Deep Learning
- Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
- A Hessian Based Complexity Measure for Deep Networks
- Understanding overfitting peaks in generalization error: Analytical risk curves for l2 and l1 penalized interpolation
- The Generalization-Stability Tradeoff in Neural Network Pruning
- Does learning require memorization? a short tale about a long tail
- Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
- Benign overfitting in linear regression
- Minimizers of the Empirical Risk and Risk Monotonicity
Related papers
- Physician-Friendly Machine Learning: A Case Study with Cardiovascular Disease Risk Prediction
- Breakdown of Machine Learning Algorithms
- Application of Machine Learning in Animal Disease Analysis and Prediction
- Machine Learning Techniques for the Management of Diseases: A Paper Review
- Artificial Intelligence, Machine Learning, and Medicine: A Little Background Goes a Long Way Toward Understanding.
- A Comprehensive Review of Various Machine Learning Techniques
- Machine Learning and Deep Learning applications-a vision using the SPSS Method
- Machine Learning and Deep Learning for Maize Leaf Disease Classification: A Review
- Improving Clinical Prediction of Later Occurrence of Breast Cancer Metastasis Using Deep Learning and Machine Learning with Grid Search