From Predictive Algorithms to Automatic Generation of Anomalies

Machine learning algorithms can find predictive signals that researchers fail to notice; yet they are notoriously hard-to-interpret. How can we extract theoretical insights from these black boxes? History provides a clue. Facing a similar problem – how to extract theoretical insights from their intu...

Full description

Saved in:
Bibliographic Details
Published in:NBER Working Paper Series
Main Authors: Mullainathan, Sendhil, Rambachan, Ashesh
Format: Paper
Language:English
Published: Cambridge National Bureau of Economic Research, Inc 01.05.2024
Subjects:
Algorithms
Expected utility
ISSN:0898-2937
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Machine learning algorithms can find predictive signals that researchers fail to notice; yet they are notoriously hard-to-interpret. How can we extract theoretical insights from these black boxes? History provides a clue. Facing a similar problem – how to extract theoretical insights from their intuitions – researchers often turned to “anomalies:” constructed examples that highlight flaws in an existing theory and spur the development of new ones. Canonical examples include the Allais paradox and the Kahneman-Tversky choice experiments for expected utility theory. We suggest anomalies can extract theoretical insights from black box predictive algorithms. We develop procedures to automatically generate anomalies for an existing theory when given a predictive algorithm. We cast anomaly generation as an adversarial game between a theory and a falsifier, the solutions to which are anomalies: instances where the black box algorithm predicts - were we to collect data - we would likely observe violations of the theory. As an illustration, we generate anomalies for expected utility theory using a large, publicly available dataset on real lottery choices. Based on an estimated neural network that predicts lottery choices, our procedures recover known anomalies and discover new ones for expected utility theory. In incentivized experiments, subjects violate expected utility theory on these algorithmically generated anomalies; moreover, the violation rates are similar to observed rates for the Allais paradox and Common ratio effect.
AbstractList Machine learning algorithms can find predictive signals that researchers fail to notice; yet they are notoriously hard-to-interpret. How can we extract theoretical insights from these black boxes? History provides a clue. Facing a similar problem – how to extract theoretical insights from their intuitions – researchers often turned to “anomalies:” constructed examples that highlight flaws in an existing theory and spur the development of new ones. Canonical examples include the Allais paradox and the Kahneman-Tversky choice experiments for expected utility theory. We suggest anomalies can extract theoretical insights from black box predictive algorithms. We develop procedures to automatically generate anomalies for an existing theory when given a predictive algorithm. We cast anomaly generation as an adversarial game between a theory and a falsifier, the solutions to which are anomalies: instances where the black box algorithm predicts - were we to collect data - we would likely observe violations of the theory. As an illustration, we generate anomalies for expected utility theory using a large, publicly available dataset on real lottery choices. Based on an estimated neural network that predicts lottery choices, our procedures recover known anomalies and discover new ones for expected utility theory. In incentivized experiments, subjects violate expected utility theory on these algorithmically generated anomalies; moreover, the violation rates are similar to observed rates for the Allais paradox and Common ratio effect.
Author Rambachan, Ashesh
Mullainathan, Sendhil
Author_xml – sequence: 1
  givenname: Sendhil
  surname: Mullainathan
  fullname: Mullainathan, Sendhil
– sequence: 2
  givenname: Ashesh
  surname: Rambachan
  fullname: Rambachan, Ashesh
BookMark eNotjU1LAzEYhHOo0A_rbwh4Xk3y7maTkyzFtkJBD72XbD40ZTfRJKt_3wVlBuZhDjNrtAgxWITuKHkAEPzxB1jN2AKtiJCiYhLaJVrnfCWECUHoCj3tUxzxW7LG6-K_Le6G95h8-RgzLhF3U4mjKl7jgw02zRQDjg53Ya4Hb_MtunFqyHb7nxt03j-fd8fq9Hp42XWnKtRtWynaUKOt5jXnchbRHCRXTW-U6alTpp4NUoM00lnVCAEKeuGo0pz2XMMG3f_Nfqb4NdlcLtc4pTA_XoA0lMqG1S38AmMkSVU
ContentType Paper
Copyright Copyright National Bureau of Economic Research, Inc. 2024
Copyright_xml – notice: Copyright National Bureau of Economic Research, Inc. 2024
DOI 10.3386/w32422
DatabaseTitleList
DeliveryMethod fulltext_linktorsrc
Discipline Economics
Genre Working Paper/Pre-Print
GroupedDBID -~X
7WY
8FL
AAWTO
ABDNZ
AFKRA
ALMA_UNASSIGNED_HOLDINGS
BENPR
F5P
K60
K6~
M0C
NBR
YNT
ID FETCH-LOGICAL-n477-a151dcec646696960c6396a5bdadb1fad4ad439c39d9fea5883a3b8f1ac61b6c3
ISSN 0898-2937
IngestDate Sat Oct 25 08:22:13 EDT 2025
IsPeerReviewed false
IsScholarly false
Language English
LinkModel OpenURL
MergedId FETCHMERGED-LOGICAL-n477-a151dcec646696960c6396a5bdadb1fad4ad439c39d9fea5883a3b8f1ac61b6c3
Notes SourceType-Working Papers-1
ObjectType-Working Paper/Pre-Print-1
content type line 50
PQID 3051195247
PQPubID 2036197
ParticipantIDs proquest_journals_3051195247
PublicationCentury 2000
PublicationDate 20240501
PublicationDateYYYYMMDD 2024-05-01
PublicationDate_xml – month: 05
  year: 2024
  text: 20240501
  day: 01
PublicationDecade 2020
PublicationPlace Cambridge
PublicationPlace_xml – name: Cambridge
PublicationTitle NBER Working Paper Series
PublicationYear 2024
Publisher National Bureau of Economic Research, Inc
Publisher_xml – name: National Bureau of Economic Research, Inc
SSID ssj0028801
Score 1.7211838
Snippet Machine learning algorithms can find predictive signals that researchers fail to notice; yet they are notoriously hard-to-interpret. How can we extract...
SourceID proquest
SourceType Aggregation Database
SubjectTerms Algorithms
Expected utility
Title From Predictive Algorithms to Automatic Generation of Anomalies
URI https://www.proquest.com/docview/3051195247
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Na9tAEF2aOJDemqYlaV2jQ-kliNqSrNWeih1sckhUYxTqnsx-qTY4kivJrX9-Z1ZfJoGSHgpCiBVokd5q9u3smxlCPlKY81SgpQ1kNLY9PgxsPpTU9iVwDRjU2hHSFJugYRgsFmxWVabLTTkBmiTBfs-2_xVqaAOwMXT2H-BuHgoNcA2gwxlgh_OzgJ9iwMgsww0YIwsabX6k2bpYPZhcDqNdkZZZWsuE0zVhHCXQvKkVhRVbDceTee1OB6651RnalgPZ4R0sYDH8Cv3vxo-qE7VqRRtz_iC4rO6N8pXOV4deBsdrNX2HSbo3V2PUyu-M6rEKnG4kgk_Em_2AYYRHmdTlsamGpTF6DX4joXPayajegA-_Lqf3t7fLaLKIPm1_2lgmDLfTq5opR6Tj0CFDM3bXv25W2GCMTInEuuOymBR29bns6MmMa2hE9Ip0zEc8Iy908pqc1u-Wn5MviJrVoma1qFlFajWoWS1qVhpbDWpvSDSdRNc3dlXbwk48Sm0OREtJLX3P9zE_UV8CU_T5UCiuxCDmyoPDZdJlisUa_qDA5a4I4gGX_kD40n1LjpM00RfEAn7LGBWO6ylcPgIB7SPnVyIQPpYbuyTd-pWX1TDNl2DlMdef49F3f7_9nrxsh0OXHBfZTn8gJ_JXsc6zHjmi3773SGc8CWfznsHiD0NzPIM
linkProvider ProQuest
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+Predictive+Algorithms+to+Automatic+Generation+of+Anomalies&rft.jtitle=NBER+Working+Paper+Series&rft.au=Mullainathan%2C+Sendhil&rft.au=Rambachan%2C+Ashesh&rft.date=2024-05-01&rft.pub=National+Bureau+of+Economic+Research%2C+Inc&rft.issn=0898-2937&rft_id=info:doi/10.3386%2Fw32422&rft.externalDBID=NO_FULL_TEXT
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0898-2937&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0898-2937&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0898-2937&client=summon