Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations

Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers reverse-mode AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on hig...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of ACM on programming languages Vol. 7; no. POPL; pp. 1573 - 1600
Main Authors: Smeding, Tom J., Vákár, Matthijs I. L.
Format: Journal Article
Language:English
Published: New York, NY, USA ACM 09.01.2023
Subjects:
Automatic differentiation
Correctness
Functional languages
Mathematics of computing
Software and its engineering
functional programming
automatic differentiation
source transformation
ISSN:2475-1421, 2475-1421
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers reverse-mode AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis used a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of linear factoring to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. We demonstrate the use of our technique by providing a practical implementation that differentiates most of Haskell98.
ISSN:2475-1421
2475-1421
DOI:10.1145/3571247