ImageNet Training in Minutes
Explore this paper's citation graph
Summary
This paper empirically evaluates the effectiveness on two neural networks: AlexNet and ResNet-50 trained with the ImageNet-1k dataset while preserving the state-of-the-art test accuracy, and uses large batch size, powered by the Layer-wise Adaptive Rate Scaling (LARS) algorithm, for efficient usage of massive computing resources.
- Type
- preprint
- Published
- 2017-09-14
- Cited by
- 470
- References
- 35
- Access
- Open access
- OpenAlex
- https://openalex.org/W2755682530
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:20425665
Keywords
Bottleneck, Computer science, Residual neural network, Training (meteorology), Epoch (astronomy)
References
- Top500 Supercomputer Sites
- On Global Combine Operations
- One weird trick for parallelizing convolutional neural networks
- A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard
- On parallelizability of stochastic gradient descent for speech DNNS
- ImageNet: A large-scale hierarchical image database
- Building high-level features using large scale unsupervised learning
- Optimization of Collective Communication Operations in MPICH
- Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Deep learning with COTS HPC systems
- Large Scale Distributed Deep Networks
- FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters
- Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin
- Deep Residual Learning for Image Recognition
- Deep Learning in Finance
- Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
- Revisiting Distributed Synchronous SGD
- 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs
- Asynchrony begets momentum, with an application to deep learning
Cited by
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
- The Case for Learned Index Structures
- Integrated Model and Data Parallelism in Training Neural Networks
- Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes
- Integrated Model, Batch, and Domain Parallelism in Training Neural Networks
- MiMatrix: A Massively Distributed Deep Learning Framework on a Petascale High-density Heterogeneous Cluster
- GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
- High Throughput Synchronous Distributed Stochastic Gradient Descent
- Productivity, Portability, Performance: Data-Centric Python
- Understanding and Controlling User Linkability in Decentralized Learning
- Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark
- Neural Inverse Rendering for General Reflectance Photometric Stereo
- RAPA-ConvNets: Modified Convolutional Networks for Accelerated Training on Architectures With Analog Arrays
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
- Optimization of hybrid parallel application execution in heterogeneous high performance computing systems considering execution time and power consumption
- Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
- Bayesian Distributed Stochastic Gradient Descent
- FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning
Related papers
- Simulation test bed for manufacturing analysis: comparison of bottleneck detection methods for AGV systems
- Comparison of bottleneck detection methods for AGV systems
- Data-driven approach for diagnostic analysis of dynamic bottlenecks in serial manufacturing systems
- Identification and characteristics analysis of bottlenecks on urban expressways based on floating car data
- Reliable Shop Floor Bottleneck Detection for Flow Lines through Process and Inventory Observations
- On the Epoch Features of Socialist Theoretical System with Chinese Characters
- Automatic Identification of Diatom Morphology using Deep Learning
- NeSSA: Near-Storage Data Selection for Accelerated Machine Learning Training