Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations

Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers reverse-mode AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on hig...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Proceedings of ACM on programming languages Jg. 7; H. POPL; S. 1573 - 1600
Hauptverfasser: Smeding, Tom J., Vákár, Matthijs I. L.
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York, NY, USA ACM 09.01.2023
Schlagworte:
Automatic differentiation
Correctness
Functional languages
Mathematics of computing
Software and its engineering
functional programming
automatic differentiation
source transformation
ISSN:2475-1421, 2475-1421
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers reverse-mode AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis used a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of linear factoring to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. We demonstrate the use of our technique by providing a practical implementation that differentiates most of Haskell98.
ISSN:2475-1421
2475-1421
DOI:10.1145/3571247