Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Keywords

Differentiable function, Estimator, Computation, Context (archaeology), Computer science

References

• Unsupervised Feature Learning and Deep Learning: A Review and New PerspectivesYoshua Bengio, Aaron Courville, Pascal Vincent — 2012
• Learning representations by back-propagating errorsDavid E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams — 1986
• Rectified Linear Units Improve Restricted Boltzmann MachinesVinod Nair, Geoffrey E. Hinton — 2010
• Improving neural networks by preventing co-adaptation of feature detectorsGeoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky — 2012
• Extracting and composing robust features with denoising autoencodersPascal Vincent, Hugo Larochelle, Yoshua Bengio — 2008
• Gradient learning in spiking neural networks by dynamic perturbation of conductances.Ila Fiete, H. Sebastian Seung — 2006
• Hierarchical Recurrent Neural Networks for Long-Term DependenciesSalah El Hihi, Yoshua Bengio — 1995
• Multivariate stochastic approximation using a simultaneous perturbation gradient approximationJames C. Spall — 1992
• Deep Sparse Rectifier Neural NetworksXavier Glorot, Antoine Bordes, Yoshua Bengio — 2011
• ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton — 2012
• Maxout NetworksIan Goodfellow, David Warde-Farley, Mehdi Mirza — 2013
• Semantic hashingRuslan Salakhutdinov, Geoffrey E. Hinton — 2009
• Deep Learning of Representations: Looking ForwardYoshua Bengio — 2013
• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement LearningRonald J. Williams — 2004
• Experimental Gerontology: Acknowledgments1993
• The Optimal Reward Baseline for Gradient-Based Reinforcement LearningLex Weaver, Nigel Tao — 2001
• Maxout NetworksIan Goodfellow, David Warde-Farley, Mehdi Mirza — 2013

Cited by

• Deep Directed Generative AutoencodersSherjil Ozair, Yoshua Bengio — 2014
• Techniques for Learning Binary Stochastic Feedforward Neural NetworksTapani Raiko, Mathias Berglund, Guillaume Alain — 2014
• Regularizing DNN acoustic models with Gaussian stochastic neuronsHao Zhang, Yajie Miao, Florian Metze — 2015
• Learning Factored Representations in a Deep Mixture of ExpertsDavid Eigen, Marc’Aurelio Ranzato, Ilya Sutskever — 2013
• Gradient Estimation Using Stochastic Computation GraphsJohn Schulman, Nicolas Heess, Théophane Weber — 2015
• Describing Multimedia Content Using Attention-Based Encoder-Decoder NetworksKyunghyun Cho, Aaron Courville, Yoshua Bengio — 2015
• Deep AutoRegressive NetworksKarol Gregor, Ivo Danihelka, Andriy Mnih — 2013
• Dynamic Capacity NetworksAmjad Almahairi, Nicolas Ballas, Tim Cooijmans — 2015
• Behavioral plasticity through the modulation of switch neuronsVassilis Vassiliades, Chris Christodoulou — 2016
• Iterative Refinement of Approximate Posterior for Training Directed Belief NetworksR Devon Hjelm, Kyunghyun Cho, Jun‐Young Chung — 2015
• Natural Language Understanding with Distributed RepresentationKyunghyun Cho — 2015
• Conditional Computation in Neural Networks for faster modelsEmmanuel Bengio, Pierre‐Luc Bacon, Joëlle Pineau — 2015
• Studies in reinforcement learning and adaptive neural networksΒασίλης Κ. Βασιλειάδης, Vassilis Vassiliades — 2015
• Compressing Word EmbeddingsMartin Andrews — 2015
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep LearningKyunghyun Cho, Yoshua Bengio — 2014
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth GradientsShuchang Zhou, Yuxin Wu, Zekun Ni — 2016
• Minimalist Regression Network with Reinforced Gradients and Weighted Estimates: a Case Study on Parameters Estimation in Automated WeldingSoheil Keshmiri — 2016
• A Deep Learning Framework of Quantized Compressed Sensing for Wireless Neural RecordingBiao Sun, Hui Feng, Kefan Chen — 2016
• Hierarchical Multiscale Recurrent Neural NetworksJun‐Young Chung, Sungjin Ahn, Yoshua Bengio — 2016
• Discrete Variational AutoencodersJason Tyler Rolfe — 2016

Related papers

• Continuous Differentiability in the Context of Generalized Approach to DifferentiabilityNikola Koceiç Bilan, Snježana Braić — 2023
• On binary function continuity,differentiability and may differentiabilityLU Yi-qing — 2012
• The Complexity Of Nowhere Differentiable Continuous FunctionsT. I. Ramsamujh — 1989
• On derivatives of fuzzy multi-dimensional mappings and applications under generalized differentiabilityM. Miri Karbasaki, M. R. Balooch Shahryari, O. Sedaghatfar — 2021
• On differentiability of Sobolev functions with respect to the Sobolev normVladimir Gol’dshtein, Paz Hashash, Alexander Ukhlov — 2022
• There exist no gaps between Gevrey differentiable and nowhere Gevrey differentiableSoon‐Yeong Chung, Jaeyoung Chung — 2004
• A Necessary and Sufficient Condition on the Differentiability of Binary FunctionsXU Xiao-li — 2009