Perturbed Iterate Analysis for Asynchronous Stochastic Optimization
Explore this paper's citation graph
Summary
Using the perturbed iterate framework, this work provides new analyses of the Hogwild! algorithm and asynchronous stochastic coordinate descent, that are simpler than earlier analyses, remove many assumptions of previous models, and in some cases yield improved upper bounds on the convergence rates.
- Type
- preprint
- Published
- 2015-07-24
- Cited by
- 252
- References
- 45
- Access
- Open access
- OpenAlex
- https://openalex.org/W1843828865
- Semantic Scholar
- https://api.semanticscholar.org/CorpusID:3876489
Keywords
Asynchronous communication, Computer science, Convergence (economics), Stochastic optimization, Stochastic gradient descent
References
- A Distributed, Asynchronous and Incremental Algorithm for Nonconvex Optimization: An ADMM Based Approach
- Parallel Correlation Clustering on Big Graphs
- Project Adam: Building an Efficient and Scalable Deep Learning Training System
- Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms
- Parallel and distributed computation
- An Asynchronous Parallel Randomized Kaczmarz Algorithm
- Large-scale matrix factorization with distributed stochastic gradient descent
- The webgraph framework I: compression techniques
- UbiCrawler: a scalable fully distributed Web crawler
- Parallel coordinate descent methods for big data optimization
- Revisiting Asynchronous Linear Solvers: Provable Convergence Rate through Randomization
- Distributed delayed stochastic optimization
- Asynchronous Stochastic Coordinate Descent: Parallelism and Convergence Properties
- Layered label propagation: a multiresolution coordinate-free ordering for compressing social networks
- Estimation, Optimization, and Parallelism when Data is Sparse
- Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
- A fast parallel SGD for matrix factorization in shared memory systems
- NOMAD: Nonlocking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion
- Identifying suspicious URLs: an application of large-scale online learning
- Randomized Smoothing for Stochastic Optimization
Cited by
- SCOPE: Scalable Composite Optimization for Learning on Spark
- Ensuring Rapid Mixing and Low Bias for Asynchronous Gibbs Sampling
- Revisiting Distributed Synchronous SGD
- The Asynchronous PALM Algorithm for Nonsmooth Nonconvex Problems
- Adversarial Delays in Online Strongly-Convex Optimization
- Cyclades: Conflict-free Asynchronous Machine Learning
- Asynchrony begets momentum, with an application to deep learning
- Asynchronous Parallel Algorithms for Nonconvex Big-Data Optimization: Model and Convergence
- Efficient Distributed SGD with Variance Reduction
- Federated Optimization: Distributed Machine Learning for On-Device Intelligence
- Parallelizing Stochastic Approximation Through Mini-Batching and Tail-Averaging
- Asynchronous Doubly Stochastic Proximal Optimization with Variance Reduction
- Stochastic Variance Reduced Optimization for Nonconvex Sparse Learning
- Zeroth-order Asynchronous Doubly Stochastic Algorithm with Variance Reduction
- Delay-Tolerant Online Convex Optimization: Unified Analysis and Adaptive-Gradient Algorithms
- An asynchronous parallel approach to sparse recovery
- Asynchronous Parallel Algorithms for Nonconvex Big-Data Optimization. Part II: Complexity and Numerical Results
- Stochastic Optimization From Distributed Streaming Data in Rate-Limited Networks
- Asynchronous Coordinate Descent under More Realistic Assumptions
- Asynchronous Stochastic Block Coordinate Descent with Variance Reduction
Related papers
- Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
- Asynchronous Stochastic Coordinate Descent: Parallelism and Convergence Properties
- Large Scale Distributed Deep Networks
- Introductory Lectures on Convex Optimization: A Basic Course
- On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants
- Optimisation stochastique pour l'apprentissage machine à grande échelle : réduction de la variance et accélération