A Hitchhiker’s Guide to Automatic Differentiation

Philipp H. W. Hoffmann · Numerical Algorithms · 2014

Summary

An overview of some of the mathematical principles of Automatic Differentiation is provided, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator.

Type: article
Published: 2014-11-03
Cited by: 45
References: 26
Access: Open access
DOI: https://doi.org/10.1007/s11075-015-0067-6
arXiv: https://arxiv.org/abs/1411.0583
OpenAlex: https://openalex.org/W2230466458
Semantic Scholar: https://api.semanticscholar.org/CorpusID:11086575

Keywords

Automatic differentiation, Algebra over a field, Dual (grammatical number), Taylor series, Theory of computation

References

Introduction to differentiable manifoldsOk Kyung Yoon, Hong-Jong Kim — 1993
Functional Coding of Differential FormsJerzy Karczmarczuk — 1999
Evaluating derivatives - principles and techniques of algorithmic differentiation, Second EditionAndreas Griewank, Andrea Walther — 2000
Nesting forward-mode AD in a functional frameworkJeffrey Mark Siskind, Barak A. Pearlmutter — 2008
Reverse-mode AD in a functional framework: Lambda the ultimate backpropagatorBarak A. Pearlmutter, Jeffrey Mark Siskind — 2008
The Arithmetic of DifferentiationL. B. Rall — 1986
Remarks on differential algebraic approach to particle beam optics by M. BerzV. Garczynski — 1993
Heap reference analysis using access graphsUday P. Khedker, Amitabha Sanyal, Amey Karkare — 2006
Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiationAndreas Griewank — 1992
A Simply Typed λ-Calculus of Forward Automatic DifferentiationOleksandr Manzyuk — 2012
A simple automatic derivative evaluation programR. E. Wengert — 1964
A new framework for the computation of HessiansRobert M. Gower, Margarida Pinheiro Mello — 2012
Differential Algebraic Description of Beam Dynamics to Very High OrdersMartin Berz — 1988
Lazy multivariate higher-order forward-mode ADBarak A. Pearlmutter, Jeffrey Mark Siskind — 2007
Doubly Recursive Multivariate Automatic DifferentiationDan Kalman — 2002
Functional Differentiation of Computer ProgramsJerzy Karczmarczuk — 1998
DiffSharp: Automatic Differentiation LibraryAtılım Güneş Baydin, Barak A. Pearlmutter, Jeffrey Mark Siskind — 2015
A First Look at Differential AlgebraJohn H. Hubbard, Benjamin Lundell — 2011
Automatic differentiation in machine learning: a surveyAtılım Güneş Baydin, Barak A. Pearlmutter, Alexey Radul — 2015
Introduction to Differentiable ManifoldsSerge Lang — 2002

Cited by

Motor Estimation using Heterogeneous Sets of Objects in Conformal Geometric AlgebraLars Tingelstad, Olav Egeland — 2016
Automatic Multivector Differentiation and OptimizationLars Tingelstad, Olav Egeland — 2016
Automatic differentiation of hybrid models Illustrated by Diffedge Graphic Methodology. (Survey)John Masse, Clara Masse, François Ollivier — 2017
Manifold Geometry with Fast Automatic Derivatives and Coordinate Frame Semantics Checking in C++Leonid Koppel, Steven L. Waslander — 2018
Interval arithmetic, hull-consistency enforcing and algorithmic differentiation using a template-based packageBartłomiej Jacek Kubica, Jarosław Kurek — 2018
Jets and differential linear logicJames Wallbridge — 2018
On the Manifold: Representing Geometry in C++ for State EstimationLeonid Koppel — 2018
Using automatic differentiation as a general framework for ptychographic reconstruction.Saugat Kandel, S. Maddali, Marc Allain — 2019
Gradient and Hessian approximations in Derivative Free OptimizationI. D. Coope, Rachael Tappenden — 2020
A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic DifferentiationCarol Mak, C.-H. Luke Ong — 2020
Graded Automatic DifferentiationKeqin Liu — 2020
Generalised Singular Value Decomposition of dual-numbered matricesRan Gutin — 2020
Complex-Valued Vs. Real-Valued Neural Networks for Classification Perspectives: An Example on Non-Circular DataJosé Agustín Barrachina, Chengfang Ren, C. Morisseau — 2020
The Interacting Multiple Model Filter on Boxplus-ManifoldsTom L. Koller, Udo Frese — 2020
Demonstration of prospective application of the dual number automatic differentiation for uncertainty propagation in neutronic calculationsPavel M. Bokov, Suzanne Groenewald, Danniëll Botes — 2021
Higher Order Automatic Differentiation with Dual NumbersLászló Szirmay‐Kalos — 2020
A Consistent Scheme for Gradient-Based Optimization of Protein-Ligand PosesFlorian Flachsenberg, Agnes Meyder, Kai Sommer — 2020
Efficient decomposition of latent representation in generative modelsVsevolod Nikulin, Jun Tani — 2020
Diagrammatic Differentiation for Quantum Machine LearningAlexis Toumi, Richie Yeung, Giovanni de Felice — 2021
Gradient and diagonal Hessian approximations using quadratic interpolation models and aligned regular basesI. D. Coope, Rachael Tappenden — 2021

Related papers

No related papers recorded.