Large Scale Online Learning

Summary

It is argued that suitably designed online learning algorithms asymptotically outperform any batch learning algorithm in situations where training data is abundant and computing resources are comparatively scarce.

Keywords

Computer science, Online learning, Scale (ratio), Artificial intelligence, Machine learning

References

Adaptive Algorithms and Stochastic ApproximationsAlbert Benveniste, Michel Métivier, Pierre Priouret — 1990
Numerical methods for unconstrained optimization and nonlinear equationsJ. E. Dennis, Robert B. Schnabel — 1983
Natural Gradient Works Efficiently in LearningШун-ичи Амари — 1998
Foundations of the theory of learning systemsI︠a︡. Z. T︠S︡ypkin — 1973
Statistical Models in SJohn M. Chambers, Trevor Hastie, D. Pregibon — 1991
Statistical analysis of learning dynamicsNoboru Murata, S. Amari — 1999
Adaptive Method of Realizing Natural Gradient Learning for Multilayer PerceptronsШун-ичи Амари, Hyeyoung Park, Kenji Fukumizu — 2000
Numerical Methods for Unconstrained Optimization and Nonlinear EquationsJ. E. Dennis, Robert B. Schnabel — 1996

Cited by

Applications of latent variable models in modeling influence and decision makingDavid M. Blei, Sean Gerrish — 2013
A Parallel Online Regularized Least-squares Machine Learning Algorithm for Future Multi-core ProcessorsT. Pahikkala, A. Airola, Thomas Canhao Xu — 2011
Online learning of large margin hidden Markov models for automatic speech recognitionLawrence K. Saul, Chih‐Chieh Cheng — 2011
A fast online algorithm for large margin training of continuous density hidden Markov modelsChih‐Chieh Cheng, Fei Sha, Lawrence K. Saul — 2009
On the importance of initialization and momentum in deep learningIlya Sutskever, James Martens, George E. Dahl — 2013
Inference Machines: Parsing Scenes via Iterated PredictionsMunoz, Daniel — 2013
Efficient Matrix Models for Relational LearningAjit Paul Singh — 2009
Prediction Games in Infinitely Rich WorldsOmid Madani — 2007
Online learning of multi-class Support Vector MachinesXuan Tuan Trinh — 2012
Online Learning for Group LassoHaiqin Yang, Zenglin Xu, Irwin King — 2010
Efficient Radio Communication for Energy Constrained Sensor NetworksTao Liu — 2013
Cumulative distribution networks: inference, estimation and applications of graphical models for cumulative distribution functionsJim Huang — 2009
Coping with the Document Frequency Bias in Sentiment ClassificationAbdelhalim Rafrafi, Vincent Guigue, Patrick Gallinari — 2012
Towards Real-Time Image Understanding with Convolutional Networks. (Analyse sémantique des images en temps-réel avec des réseaux convolutifs)Clément Farabet — 2013
Understanding Machine Learning: From Theory to AlgorithmsShai Shalev‐Shwartz, Shai Ben-David — 2014
Scalable estimation strategies based on stochastic approximations: Classical results and new insightsPanos Toulis, Edoardo M. Airoldi — 2015
Models, Inference, and Implementation for Scalable Probabilistic Models of TextKe Zhai — 2014
Algorithmes de classification répartis sur le cloudMatthieu Durut — 2012
Supervised Generative Reconstruction: An Efficient Way To Flexibly Store and Recognize PatternsTsvi Achler — 2011
Foundations and Advances in Deep LearningKyunghyun Cho — 2014

Related papers

Adaptive Subgradient Methods for Online Learning and Stochastic OptimizationJohn C. Duchi, Elad Hazan, Yoram Singer — 2010
A Stochastic Approximation MethodHerbert Robbins, Sutton Monro — 1951
Accelerating Stochastic Gradient Descent using Predictive Variance ReductionRie Johnson, Tong Zhang — 2013
Gradient-based learning applied to document recognitionYann LeCun, Léon Bottou, Yoshua Bengio — 1998
Pegasos: primal estimated sub-gradient solver for SVMShai Shalev‐Shwartz, Yoram Singer, Nathan Srebro — 2007
Adaptive Algorithms and Stochastic ApproximationsAlbert Benveniste, Michel Métivier, Pierre Priouret — 1990
Convex OptimizationStephen Boyd, Lieven Vandenberghe — 2004
Robust Stochastic Approximation Approach to Stochastic ProgrammingArkadi Nemirovski, Anatoli Juditsky, Guanghui Lan — 2008