GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
Explore this paper's citation graph
Summary
GossipGraD can achieve perfect efficiency for these datasets and their associated neural network topologies such as GoogLeNet and ResNet50 and is able to achieve ~100% compute efficiency using 128 NVIDIA Pascal P100 GPUs - while matching the top-1 classification accuracy published in literature.
- Type
- preprint
- Published
- 2018-03-15
- Cited by
- 108
- References
- 66
- Access
- Open access
- OpenAlex
- https://openalex.org/W2790598069
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:4782856
Keywords
Gossip, Asynchronous communication, Computer science, Scalability, Asynchronous learning
References
- Project Adam: Building an Efficient and Scalable Deep Learning Training System
- Torch: a modular machine learning software library
- Deep Image: Scaling up Image Recognition
- Adaptive Algorithms and Stochastic Approximations
- One weird trick for parallelizing convolutional neural networks
- Theano: new features and speed improvements
- There goes the neighborhood: Performance degradation due to nearby jobs
- Convergence and efficiency of subgradient methods for quasiconvex minimization
- Characterizing the Influence of System Noise on Large-Scale Applications by Simulation
- A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard
- On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures
- Going deeper with convolutions
- Gradient-based learning applied to document recognition
- ImageNet Large Scale Visual Recognition Challenge
- Petuum: A New Platform for Distributed Machine Learning on Big Data
- Searching for exotic particles in high-energy physics with deep learning
- RDMA read based rendezvous protocol for MPI over InfiniBand: design alternatives and benefits
- Communication Efficient Distributed Machine Learning with the Parameter Server
- More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
- A Fast Learning Algorithm for Deep Belief Nets
Cited by
- Distributed Learning over Unreliable Networks
- Deep Learning at Scale
- Priority-based Parameter Propagation for Distributed DNN Training
- Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing
- Taming unbalanced training workloads in deep learning with partial collective operations
- Decentralized Federated Learning: A Segmented Gossip Approach
- Priority-based parameter propagation for distributed deep neural network training
- Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence
- Demystifying Parallel and Distributed Deep Learning
- A generic communication scheduler for distributed DNN training acceleration
- Edge Cloud as an Enabler for Distributed AI in Industrial IoT Applications: the Experience of the IoTwins Project
- Federated Learning With Cooperating Devices: A Consensus Approach for Massive IoT Networks
- Accelerating Deep Learning Systems via Critical Set Identification and Model Compression
- BACombo—Bandwidth-Aware Decentralized Federated Learning
- Game of Threads: Enabling Asynchronous Poisoning Attacks
- Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
- Federated Learning with Mutually Cooperating Devices: A Consensus Approach Towards Server-Less Model Optimization
- A Graph Network Model for Distributed Learning with Limited Bandwidth Links and Privacy Constraints
- A Review of Privacy Preserving Federated Learning for Private IoT Analytics
- Distributed Learning in the Nonconvex World: From batch data to streaming and beyond
Related papers
- Asynchronous Discussion Forums: Success Factors, Outcomes, Assessments, and Limitations
- Designing communities of learners for asynchronous distance education
- Synchronizing asynchronous learning - Combining synchronous and asynchronous techniques
- DDHE - SYNCHRONOUS OR ASYNCHRONOUS DIDACTIC IN HIGHER EDUCATION INSTITUTIONS AFTER COVID-19
- The Efficacy of Case Method Teaching in an Online Asynchronous Learning Environment
- PATTERNS OF LEARNER-LEARNER INTERACTION IN DISTANCE LEARNING NETWORKS