Why Does Unsupervised Pre-training Help Deep Learning?

Keywords

Artificial intelligence, Computer science, Training (meteorology), Machine learning, Deep learning

References

• Computational neuroscience : theoretical insights into brain functionP. Cisek, T. Drew, J. Kalaska — 2007
• Large-scale kernel machinesL. Bottou, O. Chapelle, D. DeCoste — 2007
• To recognize shapes, first learn to generate images.Geoffrey E. Hinton — 2007
• Sensitive periods in development : interdisciplinary perspectivesMarc H. Bornstein — 1987
• Unsupervised Learning of Probabilistic Grammar-Markov Models for Object CategoriesLong Zhu, Yuanhao Chen, Alan Yuille — 2011
• Maximum mutual information estimation of hidden Markov model parameters for speech recognitionL.R. Bahl, Peter Brown, P. de Souza — 1986
• Classification using discriminative restricted Boltzmann machinesHugo Larochelle, Yoshua Bengio — 2008
• Overtraining, regularization and searching for a minimum, with application to neural networksJonas Sjöberg, Lennart Ljung — 1995
• An empirical evaluation of deep architectures on problems with many factors of variationHugo Larochelle, Dumitru Erhan, Aaron Courville — 2007
• A global geometric framework for nonlinear dimensionality reduction.Joshua B. Tenenbaum, Vin de Silva, John Langford — 2000
• Extracting and composing robust features with denoising autoencodersPascal Vincent, Hugo Larochelle, Yoshua Bengio — 2008
• Learning Deep Architectures for AIYoshua Bengio — 2007
• Restricted Boltzmann machines for collaborative filteringRuslan Salakhutdinov, Andriy Mnih, Geoffrey E. Hinton — 2007
• Measuring Invariances in Deep NetworksIan Goodfellow, Honglak Lee, Quoc V. Le — 2009
• On the power of small-depth threshold circuitsJohan H�stad, Mikael Goldmann — 1990
• Sparse Feature Learning for Deep Belief NetworksMarc’Aurelio Ranzato, Y-Lan Boureau, Y. Le Cun — 2007
• Gradient-based learning applied to document recognitionYann LeCun, Léon Bottou, Yoshua Bengio — 1998
• Cluster Kernels for Semi-Supervised LearningOlivier Chapelle, Jason Weston, Bernhard Schölkopf — 2002
• Training Products of Experts by Minimizing Contrastive DivergenceGeoffrey E. Hinton — 2002
• A unified architecture for natural language processing: deep neural networks with multitask learningRonan Collobert, Jason Weston — 2008

Cited by

• Event Recognition Based on Deep Learning in Chinese Texts航太 渡辺, 芳昭 戸山, 守雄 松本 — 2016
• Denoising Autoencoder Self-Organizing Map (DASOM)María Luisa Domínguez Reboiras — 2018
• Semisupervised Text Classification by Variational Autoencoder廣次 堀野 — 2020
• Inductive bias for semi-supervised extreme learning machineFederica Bisio, Sergio Decherchi, Paolo Gastaldo — 2016
• Unsupervised Feature Learning and Deep Learning: A Review and New PerspectivesYoshua Bengio, Aaron Courville, Pascal Vincent — 2012
• On Random Weights and Unsupervised Feature LearningAndrew Saxe, Pang Wei Koh, Zhenghao Chen — 2011
• Parallelized Deep Neural Networks for Distributed Intelligent SystemsLance Legel, Herd, Seth, Brown, Timothy — 2013
• Data Science for Neuroscience: The Brain as Inspiration, Model and Data SourceChristopher John Häusler — 2014
• An Overview of Deep-Structured Learning for Information ProcessingLi Deng — 2011
• The student-t mixture as a natural image patch prior with application to image compressionAäron van den Oord, Benjamin Schrauwen — 2014
• Quantum Machine Learning: What Quantum Computing Means to Data MiningPéter Wittek — 2014
• Deep Machine Learning with Spatio-Temporal InferenceThomas P. Karnowski — 2012
• Domain Adaptive Neural Networks for Object RecognitionMuhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang — 2014
• viSNE and Wanderlust, two algorithms for the visualization and analysis of high-dimensional single-cell dataEl-ad David Amir — 2014
• Deep learning via Hessian-free optimizationJames Martens — 2010
• Training Restricted Boltzmann MachinesAsja Fischer — 2015
• Roles of Pre-Training and Fine-Tuning in Context-Dependent DBN-HMMs for Real-World Speech RecognitionDong Yu, Li Deng, George E. Dahl — 2010
• Object tracking: Appearance modeling and feature learningRaed Almomani — 2015
• Data and feature mixed ensemble based extreme learning machine for medical object detection and segmentationWanzheng Zhu, Weimin Huang, Zhiping Lin — 2015
• A fast and efficient pre-training method based on layer-by-layer maximum discrimination for deep neural networksSeyyede Zohreh Seyyedsalehi, Seyyed Ali Seyyedsalehi, Seyyed Ali Seyyedsalehi — 2015

Related papers

• Why Does Unsupervised Pre-training Help Deep Learning?Dumitru Erhan, Aaron Courville, Yoshua Bengio — 2010
• Deep Residual Learning for Image RecognitionKaiming He, Xiangyu Zhang, Shaoqing Ren — 2015
• A Fast Learning Algorithm for Deep Belief NetsGeoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh — 2006
• Reducing the Dimensionality of Data with Neural NetworksGeoffrey E. Hinton, Ruslan Salakhutdinov — 2006
• Representation Learning: A Review and New PerspectivesYoshua Bengio, Aaron Courville, P. M. Durai Raj Vincent — 2012
• AN EXPLORATORY STUDY TOWARDS APPLYING AND DEMYSTIFYING DEEP LEARNING CLASSIFICATION ON BEHAVIORAL BIG DATASudhir Allam — 2021
• GloVe: Global Vectors for Word RepresentationJeffrey Pennington, Richard Socher, Christopher D. Manning — 2014
• A global geometric framework for nonlinear dimensionality reduction.Joshua B. Tenenbaum, Vin de Silva, John Langford — 2000
• Unsupervised feature learning-based encoder and adversarial networksEndang Suryawati, Hilman F. Pardede, Vicky Zilvan — 2021