Demystifying differentiable programming: shift/reset the penultimate backpropagator

Keywords

Computer science, Differentiable function, Artificial neural network, Backpropagation, Gradient descent

References

• Torch7: A Matlab-like Environment for Machine LearningRonan Collobert, Koray Kavukcuoglu, Clément Farabet — 2011
• Librispeech: An ASR corpus based on public domain audio booksVassil Panayotov, Guoguo Chen, Daniel Povey — 2015
• Learning representations by back-propagating errorsDavid E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams — 1986
• Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral SciencesPaul J. Werbos — 1974
• Compiling Fast Partial Derivatives of Functions Given by AlgorithmsB. Speelpenning — 1980
• Analytical differentiation on a digital computerJohn F. Nolan — 1953
• Learning to Transduce with Unbounded MemoryEdward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman — 2015
• There and back againOlivier Danvy, Mayer Goldberg — 2001
• Nesting forward-mode AD in a functional frameworkJeffrey Mark Siskind, Barak A. Pearlmutter — 2008
• There and back againFrancis Jones — 1989
• Reverse-mode AD in a functional framework: Lambda the ultimate backpropagatorBarak A. Pearlmutter, Jeffrey Mark Siskind — 2008
• Applied Optimal Control: Optimization, Estimation, and ControlArthur E. Bryson, Yu‐Chi Ho, George M. Siouris — 1979
• On the momentum term in gradient descent learning algorithmsNing Qian — 1999
• Compilers and staging transformationsUlrik Jørring, William L. Scherlis — 1986
• A Steepest-Ascent Method for Solving Optimum Programming ProblemsA. E. Bryson, WALTER F. DENHAM — 1962
• Representing Control: a Study of the CPS TransformationOliver Danvy, Andrzex Filinski — 1992
• The theory and practice of first-class promptsMatthias Felleisen — 1988
• Optimizing data structures in high-level programs: new directions for extensible compilers based on stagingTiark Rompf, Arvind K. Sujeeth, Nada Amin — 2013
• Taylor expansion of the accumulated rounding errorSeppo Linnainmaa — 1976
• An Operational Foundation for Delimited Continuations in the CPS HierarchyMałgorzata Biernacka, Dariusz Biernacki, Olivier Danvy — 2005

Cited by

• k-meansNet: When k-means Meets Differentiable ProgrammingXi Peng, Joey Tianyi Zhou, Hongyuan Zhu — 2018
• AutoGraph: Imperative-style Coding with Graph-based PerformanceZhang, Guoqiang, Shen, Xipeng — 2018
• Dynamic Automatic Differentiation of GPU Broadcast KernelsJarrett Revels, Tim Besard, Valentin Churavy — 2018
• Deep learning for pedestrians: backpropagation in CNNsLaurent Boué — 2018
• Residual Reinforcement Learning for Robot ControlTobias Johannink, Shikhar Bahl, Ashvin Nair — 2018
• Relay: A High-Level IR for Deep LearningJared Roesch, Steven Lyubomirsky, Marisa Kirisame — 2019
• Sequence-to-sequence learning for machine translation and automatic differentiation for machine learning software toolsBart van Merriënboer — 2019
• A Differentiable Programming System to Bridge Machine Learning and Scientific ComputingMike Innes, Alan Edelman, Keno Fischer — 2019
• Functional probabilistic programming for scalable Bayesian modellingJ. Law, Darren J. Wilkinson — 2019
• Relay: A High-Level Compiler for Deep LearningRoesch, Jared, Lyubomirsky, Steven, Kirisame, Marisa — 2019
• Flare & Lantern: Efficiently Swapping Horses MidstreamGrégory M. Essertel, Ruby Y. Tahboub, Fei Wang — 2019
• Backpropagation in the simply typed lambda-calculus with linear negationAloïs Brunel, Damiano Mazza, Michele Pagani — 2019
• The Differentiable CurryDimitrios Vytiniotis, Dan Belov, Richard Wei — 2019
• A simple differentiable programming languageMartı́n Abadi, Gordon Plotkin — 2019
• Abstractions for Programming Graphics Processors in High-Level Programming LanguagesTim Besard — 2018
• Resilient Cyberphysical Systems and their Application Drivers: A Technology RoadmapSomali Chaterji, Parinaz Naghizadeh, Md Ashraful Alam — 2019
• Correctness of Automatic Differentiation via Diffeologies and Categorical GluingMathieu Huot, Sam Staton, Matthijs Vákár — 2020
• A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic DifferentiationCarol Mak, C.-H. Luke Ong — 2020
• Inverse design of photonic crystals through automatic differentiationMomchil Minkov, Ian A. D. Williamson, Lucio C. Andreani — 2020
• Parallel Training via Computation Graph TransformationFei Wang, Guoyang Chen, Weifeng Zhang — 2019

Related papers

• On proto-differentiability of generalized perturbation mapsG.M. Lee, N. Q. Huy — 2006
• Training Artificial Neural Networks: Backpropagation via Nonlinear OptimizationJadranka Skorin‐Kapov, Keshuang Tang — 2001
• Analisis Penurunan Gradien dengan Kombinasi Fungsi Aktivasi pada Algoritma JST untuk Pencarian Akurasi TerbaikAnjar Wanto, Jufriadif Na’am, Yuhandri Yunus — 2020
• Exponentiated backpropagation algorithm for multilayer feedforward neural networksNarayanan Srinivasan, V. Ravichandran, Kok Lim Chan — 2002
• A New Backpropagation Algorithm without Gradient DescentVarun Ranganathan, Sayee Natarajan — 2018
• Gradients without BackpropagationAtılım Güneş Baydin, Barak A. Pearlmutter, Don Syme — 2022
• Improving the Training Performance of Neural Networks by using Hybrid AlgorithmWeon-Ook Kim, Yong‐Hyun Cho, Youngil Kim — 1997
• Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient MethodErik M. J. Johansson, Farid Dowla, Dennis M. Goodman — 1991
• Modelling of One-directional Functionally Graded Circular Plates with Artificial Neural NetworkDidem Çakır, Munise Didem Demirbaş — 2019