Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming

We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipu...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings of ACM on programming languages Vol. 5; no. ICFP; pp. 1 - 29
Main Authors: Paszke, Adam, Johnson, Daniel D., Duvenaud, David, Vytiniotis, Dimitrios, Radul, Alexey, Johnson, Matthew J., Ragan-Kelley, Jonathan, Maclaurin, Dougal
Format: Journal Article
Language:English
Published: 01.08.2021
ISSN:2475-1421, 2475-1421
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, "pointful" style with little performance penalty.
AbstractList We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arrays as eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, "pointful" style with little performance penalty.
Author Duvenaud, David
Vytiniotis, Dimitrios
Ragan-Kelley, Jonathan
Paszke, Adam
Radul, Alexey
Maclaurin, Dougal
Johnson, Daniel D.
Johnson, Matthew J.
Author_xml – sequence: 1
  givenname: Adam
  surname: Paszke
  fullname: Paszke, Adam
  organization: Google Research, Poland
– sequence: 2
  givenname: Daniel D.
  surname: Johnson
  fullname: Johnson, Daniel D.
  organization: Google Research, Canada
– sequence: 3
  givenname: David
  surname: Duvenaud
  fullname: Duvenaud, David
  organization: University of Toronto, Canada
– sequence: 4
  givenname: Dimitrios
  surname: Vytiniotis
  fullname: Vytiniotis, Dimitrios
  organization: DeepMind, UK
– sequence: 5
  givenname: Alexey
  surname: Radul
  fullname: Radul, Alexey
  organization: Google Research, USA
– sequence: 6
  givenname: Matthew J.
  surname: Johnson
  fullname: Johnson, Matthew J.
  organization: Google Research, USA
– sequence: 7
  givenname: Jonathan
  surname: Ragan-Kelley
  fullname: Ragan-Kelley, Jonathan
  organization: Massachusetts Institute of Technology, USA
– sequence: 8
  givenname: Dougal
  surname: Maclaurin
  fullname: Maclaurin, Dougal
  organization: Google Research, USA
BookMark eNplkE1Lw0AYhBepYK3Fv7A3T9F3P_LlTYpWoeBFz-FN8m5dSbJhdyv239vQHkRPM4dnhmEu2WxwAzF2LeBWCJ3eKZ2rtFRnbC51niZCSzH75S_YMoRPABCl0oUq56xZU4x22PLoePwgPjo7xHtuh5a-eaAYOA4tH9Fj11FnQ5-MngL5rymDu-haaww3zh-TZtdx9B73fPRu67HvD9wVOzfYBVqedMHenx7fVs_J5nX9snrYJI1Mi5hITI3STZlplDVImhamWZaTLApdywIL0tBCLaEGMBlCXssWUGijSROQWrDk2Nt4F4InUzU2YrRuiB5tVwmoppOq00kH_uYPP3rbo9__I38AiNloqQ
CitedBy_id crossref_primary_10_1145_3747524
crossref_primary_10_1145_3498717
crossref_primary_10_1145_3618353
crossref_primary_10_1017_S0956796825100051
crossref_primary_10_1145_3591260
crossref_primary_10_1145_3528223_3530099
crossref_primary_10_1145_3592622
crossref_primary_10_1145_3563324
crossref_primary_10_1145_3632878
crossref_primary_10_1145_3674651
crossref_primary_10_1016_j_scico_2024_103086
crossref_primary_10_1145_3571236
crossref_primary_10_1145_3665643
Cites_doi 10.1145/3355089.3356506
10.1137/141000671
10.1145/3236765
10.1145/1926354.1926358
10.1145/2935323.2935326
10.1145/3197517.3201383
10.1145/3337821.3337906
10.1145/178243.178246
10.1109/SC41405.2020.00101
10.1145/3341701
10.1137/1.9780898717761
10.1007/978-3-540-89330-1_10
10.1017/S095679681900008X
10.1017/S0956796802004331
10.1142/S0129626412500107
10.1145/2500365.2500595
10.1109/CGO.2017.7863730
10.1109/IISWC.2009.5306797
10.1145/2499370.2462176
10.1145/3428194
10.1145/3122948.3122952
10.4204/eptcs.153.8
10.1007/978-3-642-16478-1_5
10.1155/1992/717832
10.1007/978-0-387-09766-4_190
10.1145/2450153.2450158
10.1145/2847538.2847541
10.1007/978-3-030-18506-0_7
10.1007/978-3-642-54833-8_3
10.1145/3211346.3211348
10.1017/S0956796806006034
10.1145/3292500.3330756
10.1145/3062341.3062354
10.5555/3122009.3242010
10.1145/1863543.1863588
10.5555/865114
10.1145/2034773.2034811
10.1145/1330017.1330018
10.1145/1375581.1375595
10.1038/s41586-020-2649-2
ContentType Journal Article
DBID AAYXX
CITATION
DOI 10.1145/3473593
DatabaseName CrossRef
DatabaseTitle CrossRef
DatabaseTitleList CrossRef
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISSN 2475-1421
EndPage 29
ExternalDocumentID 10_1145_3473593
GroupedDBID AAKMM
AAYFX
AAYXX
ACM
AEFXT
AEJOY
AIKLT
AKRVB
ALMA_UNASSIGNED_HOLDINGS
CITATION
GUFHI
LHSKQ
M~E
OK1
ROL
ID FETCH-LOGICAL-c258t-2a5f34c964a2b02e34835667e2884b28a8e40d0b20b00f6a07b2d0a14f4e4e0e3
ISICitedReferencesCount 26
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000687133800028&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 2475-1421
IngestDate Tue Nov 18 21:33:43 EST 2025
Sat Nov 29 07:52:54 EST 2025
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue ICFP
Language English
LinkModel OpenURL
MergedId FETCHMERGED-LOGICAL-c258t-2a5f34c964a2b02e34835667e2884b28a8e40d0b20b00f6a07b2d0a14f4e4e0e3
OpenAccessLink https://dl.acm.org/doi/pdf/10.1145/3473593
PageCount 29
ParticipantIDs crossref_citationtrail_10_1145_3473593
crossref_primary_10_1145_3473593
PublicationCentury 2000
PublicationDate 2021-08-01
PublicationDateYYYYMMDD 2021-08-01
PublicationDate_xml – month: 08
  year: 2021
  text: 2021-08-01
  day: 01
PublicationDecade 2020
PublicationTitle Proceedings of ACM on programming languages
PublicationYear 2021
References e_1_2_2_4_1
e_1_2_2_24_1
e_1_2_2_49_1
Abadi Martín (e_1_2_2_1_1) 2016
e_1_2_2_6_1
e_1_2_2_22_1
e_1_2_2_20_1
e_1_2_2_2_1
e_1_2_2_41_1
Hu Yuanming (e_1_2_2_23_1) 2020
Maclaurin Dougal (e_1_2_2_32_1) 2014
e_1_2_2_43_1
e_1_2_2_8_1
e_1_2_2_28_1
e_1_2_2_45_1
e_1_2_2_47_1
Vasilache Nicolas (e_1_2_2_54_1) 2018
Paszke Adam (e_1_2_2_39_1) 2019
e_1_2_2_13_1
e_1_2_2_38_1
e_1_2_2_11_1
Frostig Roy (e_1_2_2_14_1) 2021
e_1_2_2_30_1
e_1_2_2_19_1
e_1_2_2_53_1
e_1_2_2_17_1
e_1_2_2_34_1
e_1_2_2_15_1
Milner Robin (e_1_2_2_36_1) 1814
e_1_2_2_25_1
e_1_2_2_48_1
e_1_2_2_5_1
e_1_2_2_7_1
Innes Michael (e_1_2_2_26_1) 2018
e_1_2_2_21_1
e_1_2_2_40_1
Bergstra James (e_1_2_2_3_1) 2010
e_1_2_2_42_1
e_1_2_2_9_1
e_1_2_2_29_1
e_1_2_2_44_1
e_1_2_2_46_1
Iverson Kenneth E. (e_1_2_2_27_1)
e_1_2_2_37_1
e_1_2_2_12_1
Bradbury James (e_1_2_2_10_1) 2018
e_1_2_2_52_1
e_1_2_2_31_1
e_1_2_2_18_1
e_1_2_2_33_1
e_1_2_2_16_1
e_1_2_2_35_1
e_1_2_2_50_1
Stratton J. A. (e_1_2_2_51_1) 2012
References_xml – ident: e_1_2_2_24_1
  doi: 10.1145/3355089.3356506
– ident: e_1_2_2_5_1
  doi: 10.1137/141000671
– ident: e_1_2_2_13_1
  doi: 10.1145/3236765
– ident: e_1_2_2_11_1
  doi: 10.1145/1926354.1926358
– ident: e_1_2_2_20_1
  doi: 10.1145/2935323.2935326
– ident: e_1_2_2_31_1
  doi: 10.1145/3197517.3201383
– ident: e_1_2_2_25_1
  doi: 10.1145/3337821.3337906
– ident: e_1_2_2_29_1
  doi: 10.1145/178243.178246
– volume-title: Parboil: A Revised Benchmark Suite for Scientific and Commercial Throughput Computing.
  year: 2012
  ident: e_1_2_2_51_1
– ident: e_1_2_2_19_1
  doi: 10.1109/SC41405.2020.00101
– ident: e_1_2_2_45_1
– ident: e_1_2_2_47_1
  doi: 10.1145/3341701
– ident: e_1_2_2_15_1
  doi: 10.1137/1.9780898717761
– ident: e_1_2_2_4_1
– volume-title: Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.
  year: 2018
  ident: e_1_2_2_10_1
– volume-title: AutoML workshop.
  year: 2014
  ident: e_1_2_2_32_1
– ident: e_1_2_2_41_1
  doi: 10.1007/978-3-540-89330-1_10
– ident: e_1_2_2_33_1
  doi: 10.1017/S095679681900008X
– ident: e_1_2_2_42_1
  doi: 10.1017/S0956796802004331
– ident: e_1_2_2_16_1
  doi: 10.1142/S0129626412500107
– ident: e_1_2_2_35_1
  doi: 10.1145/2500365.2500595
– ident: e_1_2_2_50_1
  doi: 10.1109/CGO.2017.7863730
– ident: e_1_2_2_12_1
  doi: 10.1109/IISWC.2009.5306797
– ident: e_1_2_2_44_1
  doi: 10.1145/2499370.2462176
– ident: e_1_2_2_9_1
  doi: 10.1145/3428194
– ident: e_1_2_2_28_1
  doi: 10.1145/3122948.3122952
– ident: e_1_2_2_30_1
  doi: 10.4204/eptcs.153.8
– ident: e_1_2_2_34_1
  doi: 10.1007/978-3-642-16478-1_5
– volume-title: DiffTaichi: Differentiable Programming for Physical Simulation. In International Conference on Learning Representations. https://openreview.net/forum?id=B1eB5xSFvr
  year: 2020
  ident: e_1_2_2_23_1
– ident: e_1_2_2_6_1
  doi: 10.1155/1992/717832
– volume-title: The Definition of Standard ML
  year: 1814
  ident: e_1_2_2_36_1
– volume-title: High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019
  year: 2019
  ident: e_1_2_2_39_1
– volume-title: Proceedings of the Python for scientific computing conference (SciPy). 4, 1–7.
  year: 2010
  ident: e_1_2_2_3_1
– ident: e_1_2_2_49_1
  doi: 10.1007/978-0-387-09766-4_190
– ident: e_1_2_2_18_1
  doi: 10.1145/2450153.2450158
– ident: e_1_2_2_38_1
  doi: 10.1145/2847538.2847541
– volume-title: A Programming Language
  ident: e_1_2_2_27_1
– ident: e_1_2_2_22_1
  doi: 10.1007/978-3-030-18506-0_7
– ident: e_1_2_2_48_1
  doi: 10.1007/978-3-642-54833-8_3
– ident: e_1_2_2_46_1
  doi: 10.1145/3211346.3211348
– ident: e_1_2_2_43_1
  doi: 10.1017/S0956796806006034
– ident: e_1_2_2_53_1
  doi: 10.1145/3292500.3330756
– volume-title: Don’t Unroll Adjoint: Differentiating SSA-Form Programs. CoRR, abs/1810.07951
  year: 2018
  ident: e_1_2_2_26_1
– volume-title: Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions. arxiv:1802.04730.
  year: 2018
  ident: e_1_2_2_54_1
– ident: e_1_2_2_21_1
  doi: 10.1145/3062341.3062354
– ident: e_1_2_2_2_1
  doi: 10.5555/3122009.3242010
– ident: e_1_2_2_37_1
  doi: 10.1145/1863543.1863588
– volume-title: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI’16)
  year: 2016
  ident: e_1_2_2_1_1
– ident: e_1_2_2_7_1
  doi: 10.5555/865114
– volume-title: POPL 2021 workshop on Languages for Inference.
  year: 2021
  ident: e_1_2_2_14_1
– ident: e_1_2_2_52_1
  doi: 10.1145/2034773.2034811
– ident: e_1_2_2_40_1
  doi: 10.1145/1330017.1330018
– ident: e_1_2_2_8_1
  doi: 10.1145/1375581.1375595
– ident: e_1_2_2_17_1
  doi: 10.1038/s41586-020-2649-2
SSID ssj0001934839
Score 2.3992248
Snippet We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and...
SourceID crossref
SourceType Enrichment Source
Index Database
StartPage 1
Title Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming
Volume 5
WOSCitedRecordID wos000687133800028&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2475-1421
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001934839
  issn: 2475-1421
  databaseCode: M~E
  dateStart: 20170101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Lb9QwELaWwoELlJdaXvIBcalSEsdJbCQOq20LSLTaQ0G9rZzYllbKJqtstmo58K_6_zp-JE2XSsCBSxRZdhRnPs2MJ9_MIPSOgZEjaayDyHAHqYRzioh1FGiW80hRydPCFnH9lp2csLMzPh2NrrpcmPMyqyp2ccGX_1XUMAbCNqmz_yDu_qEwAPcgdLiC2OH6V4L_rByV2TuVy3ruwn-2LuLeSrWuKrOp-V2WqpyvFoEhwxqdYfIV121tmqZY-qFda0nMTSMuOy7XorN23qed9jbQ0kLGk2PzB2IwuQ-K9v77VKx-OlbQWIrFkMXjE8Bc5vvewf6Now1aWaw3efjrTz8uYbfzunW1Eg5MvlYzr29FM0jUc-m80iM0S4KIuqzpfXXHmNfayQCcXydH04ESjgbW3IVTfrcT1JTUiE3fZdeg8XYl7g0L2fMWXRZ3MvML76H7JEu4YRIe_xqE9nhMme1h17-5S9c2az_4tQM_aODQnG6jR_4kgscOQU_QSFVP0eOuywf2Sv8ZKjygcFtjABS2oPiILZywgRMGOOG74YQ7OGGAE-7ghC2c8AAhz9H3o8PTyZfAt-YICpKwNiAi0TEteEoFyUOizIbhYJApwhjNCRNM0VCGOQlBretUhFlOZCgiqqmiKlTxC7RV1ZXaQZhLSnkqmJaGBZArETMtojQnueRS6ngXve8-1KzwdetN-5RytiGNXYT7iUtXqmVzyss_T3mFHt7g8jXaapu1eoMeFOeA4-atFfQ1RgCLEA
linkProvider ISSN International Centre
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Getting+to+the+point%3A+index+sets+and+parallelism-preserving+autodiff+for+pointful+array+programming&rft.jtitle=Proceedings+of+ACM+on+programming+languages&rft.au=Paszke%2C+Adam&rft.au=Johnson%2C+Daniel+D.&rft.au=Duvenaud%2C+David&rft.au=Vytiniotis%2C+Dimitrios&rft.date=2021-08-01&rft.issn=2475-1421&rft.eissn=2475-1421&rft.volume=5&rft.issue=ICFP&rft.spage=1&rft.epage=29&rft_id=info:doi/10.1145%2F3473593&rft.externalDBID=n%2Fa&rft.externalDocID=10_1145_3473593
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2475-1421&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2475-1421&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2475-1421&client=summon