Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks

We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which an RNN with neurons learns to integrate scalar sign...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Neural computation Ročník 33; číslo 4; s. 1
Hlavní autoři: Fanthomme, Arnaud, Monasson, Rémi
Médium: Journal Article
Jazyk:angličtina
Vydáno: United States 26.03.2021
ISSN:1530-888X, 1530-888X
On-line přístup:Zjistit podrobnosti o přístupu
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstract We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which an RNN with neurons learns to integrate scalar signals of arbitrary duration. We show, for linear, ReLU, and sigmoidal neurons, that the internal state lives close to a -dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.
AbstractList We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which an RNN with n neurons learns to integrate D(≪n) scalar signals of arbitrary duration. We show, for linear, ReLU, and sigmoidal neurons, that the internal state lives close to a D-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which an RNN with n neurons learns to integrate D(≪n) scalar signals of arbitrary duration. We show, for linear, ReLU, and sigmoidal neurons, that the internal state lives close to a D-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.
We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which an RNN with neurons learns to integrate scalar signals of arbitrary duration. We show, for linear, ReLU, and sigmoidal neurons, that the internal state lives close to a -dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.
Author Fanthomme, Arnaud
Monasson, Rémi
Author_xml – sequence: 1
  givenname: Arnaud
  surname: Fanthomme
  fullname: Fanthomme, Arnaud
  email: arnaud.fanthomme@phys.ens.fr
  organization: Laboratoire de Physique de l'Ecole Normale Supérieure PSLand CNRS UMR 8023, Sorbonne Université, 75005 Paris, France arnaud.fanthomme@phys.ens.fr
– sequence: 2
  givenname: Rémi
  surname: Monasson
  fullname: Monasson, Rémi
  email: remi.monasson@phys.ens.fr
  organization: Laboratoire de Physique de l'Ecole Normale Supérieure PSLand CNRS UMR 8023, Sorbonne Université, 75005 Paris, France remi.monasson@phys.ens.fr
BackLink https://www.ncbi.nlm.nih.gov/pubmed/33513327$$D View this record in MEDLINE/PubMed
BookMark eNpNkDtPwzAYRS1URB-wMaOMLAE_EscZUSlQqS0SD8EWOfZnFEjsYDsq_HsqUSSme4dz7nCnaGSdBYROCb4ghNNLC8pVssKEcX6AJiRnOBVCvI7-9TGahvCOMeYE50dozFhOGKPFBL2s3Da9bjqwoXFWtsla2sa4Vofkceh752OyHtrY9C18gU6WNsKbl3HHhqSxyQOowXuwMdnA4Hf6BuLW-Y9wjA6NbAOc7HOGnm8WT_O7dHV_u5xfrVLFchrTTEkt6sIIQjJuSCEzUwLhOTacCG0wz7RWjNaCZpSVZa1rDKoWBS8KY4SSdIbOf3d77z4HCLHqmqCgbaUFN4SKZoIJUjBW7tCzPTrUHeiq900n_Xf1dwb9Ae0jZNw
CitedBy_id crossref_primary_10_1155_2021_2438656
ContentType Journal Article
Copyright 2021 Massachusetts Institute of Technology.
Copyright_xml – notice: 2021 Massachusetts Institute of Technology.
DBID NPM
7X8
DOI 10.1162/neco_a_01366
DatabaseName PubMed
MEDLINE - Academic
DatabaseTitle PubMed
MEDLINE - Academic
DatabaseTitleList MEDLINE - Academic
PubMed
Database_xml – sequence: 1
  dbid: NPM
  name: PubMed
  url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed
  sourceTypes: Index Database
– sequence: 2
  dbid: 7X8
  name: MEDLINE - Academic
  url: https://search.proquest.com/medline
  sourceTypes: Aggregation Database
DeliveryMethod no_fulltext_linktorsrc
Discipline Computer Science
EISSN 1530-888X
ExternalDocumentID 33513327
Genre Journal Article
GroupedDBID ---
-~X
.4S
.DC
0R~
123
36B
4.4
6IK
AAJGR
AALMD
ABAZT
ABDBF
ABDNZ
ABIVO
ABJNI
ACGFO
ACUHS
ADMLS
AEGXH
AEILP
AENEX
AFHIN
AIAGR
ALMA_UNASSIGNED_HOLDINGS
ARCSS
AVWKF
AZFZN
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EAP
EAS
EBC
EBD
EBS
ECS
EDO
EMB
EMK
EMOBN
EPL
EPS
EST
ESX
F5P
FEDTE
FNEHJ
HZ~
I-F
IPLJI
JAVBF
MCG
MINIK
MKJ
NPM
O9-
OCL
P2P
PK0
PQQKQ
RMI
SV3
TUS
WG8
WH7
XJE
ZWS
7X8
ABVLG
AMVHM
ID FETCH-LOGICAL-c352t-4cad8b7f81146f17a4f9e1650f618df064ddc32b8242399bdb0ecb87677ff8ca2
IEDL.DBID 7X8
ISICitedReferencesCount 1
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000663433900008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 1530-888X
IngestDate Wed Oct 01 15:05:06 EDT 2025
Wed Feb 19 02:29:17 EST 2025
IsPeerReviewed true
IsScholarly true
Issue 4
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c352t-4cad8b7f81146f17a4f9e1650f618df064ddc32b8242399bdb0ecb87677ff8ca2
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
PMID 33513327
PQID 2483817339
PQPubID 23479
ParticipantIDs proquest_miscellaneous_2483817339
pubmed_primary_33513327
PublicationCentury 2000
PublicationDate 2021-03-26
PublicationDateYYYYMMDD 2021-03-26
PublicationDate_xml – month: 03
  year: 2021
  text: 2021-03-26
  day: 26
PublicationDecade 2020
PublicationPlace United States
PublicationPlace_xml – name: United States
PublicationTitle Neural computation
PublicationTitleAlternate Neural Comput
PublicationYear 2021
SSID ssj0006105
Score 2.3563745
Snippet We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals....
SourceID proquest
pubmed
SourceType Aggregation Database
Index Database
StartPage 1
Title Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
URI https://www.ncbi.nlm.nih.gov/pubmed/33513327
https://www.proquest.com/docview/2483817339
Volume 33
WOSCitedRecordID wos000663433900008&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LS8QwEA7qevDi-nZ9EcFrsG26aXMSUReF3bKIYm8lzQMWpF3t-vj5zvTBngTBS28pYTKZfpl8_T5CLngYK6h6HstVZFk45B6TMudMqdCXepiLUNXq-uMoSeI0ldO24Va1tMquJtaF2pQae-SXQRijmBzn8mr-xtA1Cm9XWwuNVdLjAGUwq6N0qRYuGgqjjzOAk17aEd9FcFnA4S5TGSqWid_BZf2RGfX_O70tstnCS3rd5MM2WbHFDul31g203cm75GVcfrFbVPZvVDnoRBUzV76aiqLPJ2ByOmmpht_W0IdWVAJzlM4K-ohdetR1oijuAcOThk1e7ZHn0d3TzT1rPRaYBui1YKFWJs4jF-Pfyc6PVOik9QG2OeHHxgFgMUbzII8Rd8EqmtyzOocSGkXOxVoF-2StKAt7SKj0lNC5p4yAhdZcSaECeBfgT19YK8MBOe9Cl0EO48WEKmz5UWXL4A3IQRP_bN6IbWScowNNEB39YfQx2QiQcuJxFogT0nOwg-0pWdefi1n1flYnBzyT6eQHLszGwg
linkProvider ProQuest
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Low-Dimensional+Manifolds+Support+Multiplexed+Integrations+in+Recurrent+Neural+Networks&rft.jtitle=Neural+computation&rft.au=Fanthomme%2C+Arnaud&rft.au=Monasson%2C+R%C3%A9mi&rft.date=2021-03-26&rft.eissn=1530-888X&rft.spage=1&rft_id=info:doi/10.1162%2Fneco_a_01366&rft_id=info%3Apmid%2F33513327&rft_id=info%3Apmid%2F33513327&rft.externalDocID=33513327
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1530-888X&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1530-888X&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1530-888X&client=summon