Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
Explore this paper's citation graph
Summary
This work builds a highly scalable deep learning training system for dense GPU clusters with three main contributions: a mixed-precision training method that significantly improves the training throughput of a single GPU without losing accuracy, an optimization approach for extremely large mini-batch size that can train CNN models on the ImageNet dataset without lost accuracy, and highly optimized all-reduce algorithms.
- Type
- preprint
- Published
- 2018-07-30
- Cited by
- 424
- References
- 31
- Access
- Open access
- OpenAlex
- https://openalex.org/W2884711234
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:51876267
Keywords
Training (meteorology), Scalability, Computer science, Artificial intelligence, Machine learning
References
- Training deep neural networks with low precision multiplications
- cuDNN: Efficient Primitives for Deep Learning
- Interprocessor collective communication library (InterCom)
- ImageNet: A large-scale hierarchical image database
- Optimization of Collective Communication Operations in MPICH
- A Simple Weight Decay Can Improve Generalization
- ImageNet classification with deep convolutional neural networks
- Deep Residual Learning for Image Recognition
- TensorFlow: a system for large-scale machine learning
- Ako: Decentralised Deep Learning with Partial Gradient Exchange
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
- Scaling SGD Batch Size to 32K for ImageNet Training
- ImageNet Training in Minutes
- Large Batch Training of Convolutional Networks
- Mixed Precision Training
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
Cited by
- Database Meets Deep Learning: Challenges and Opportunities
- SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning
- Large batch size training of neural networks with adversarial training and second-order information
- Exascale Deep Learning for Climate Analytics
- A Hitchhiker's Guide On Distributed Training of Deep Neural Networks
- Democratizing Production-Scale Distributed Deep Learning
- ImageNet/ResNet-50 Training in 224 Seconds
- Revisiting Pre-training: An Efficient Training Method for Image Classification.
- Hydra: A Peer to Peer Distributed Training & Data Collection Framework
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- Image Classification at Supercomputer Scale
- Accelerating reduction and scan using tensor core units
- On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
- Bag of Tricks for Image Classification with Convolutional Neural Networks
- Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs
- An Empirical Model of Large-Batch Training
- Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
- Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
- Crossbow: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
- TF-Replicator: Distributed Machine Learning for Researchers
Related papers
- Physician-Friendly Machine Learning: A Case Study with Cardiovascular Disease Risk Prediction
- Breakdown of Machine Learning Algorithms
- Application of Machine Learning in Animal Disease Analysis and Prediction
- Artificial Intelligence, Machine Learning, and Medicine: A Little Background Goes a Long Way Toward Understanding.
- A Comprehensive Review of Various Machine Learning Techniques
- Machine Learning and Deep Learning applications-a vision using the SPSS Method
- Machine Learning and Deep Learning for Maize Leaf Disease Classification: A Review
- Improving Clinical Prediction of Later Occurrence of Breast Cancer Metastasis Using Deep Learning and Machine Learning with Grid Search
- Machine Learning and Deep Learning Applications in Multiple Myeloma Diagnosis, Prognosis, and Treatment Selection