Continuously Constructive Deep Neural Networks
Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In this paper, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of o...
Uloženo v:
| Vydáno v: | IEEE transaction on neural networks and learning systems Ročník 31; číslo 4; s. 1124 - 1133 |
|---|---|
| Hlavní autoři: | , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
United States
IEEE
01.04.2020
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Témata: | |
| ISSN: | 2162-237X, 2162-2388, 2162-2388 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In this paper, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of our approach lies in our parameterization, where the depth, or additional complexity, is encapsulated continuously in the parameter space through control parameters that add additional complexity. We propose two methods. In tunnel networks, this selection is done at the level of a hidden unit, and in budding perceptrons, this is done at the level of a network layer; updating this control parameter introduces either another hidden unit or layer. We show the effectiveness of our methods on the synthetic two-spiral data and on three real data sets of MNIST, MIRFLICKR, and CIFAR, where we see that our proposed methods, with the same set of hyperparameters, can correctly adjust the network complexity to the task complexity. |
|---|---|
| AbstractList | Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In this paper, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of our approach lies in our parameterization, where the depth, or additional complexity, is encapsulated continuously in the parameter space through control parameters that add additional complexity. We propose two methods. In tunnel networks, this selection is done at the level of a hidden unit, and in budding perceptrons, this is done at the level of a network layer; updating this control parameter introduces either another hidden unit or layer. We show the effectiveness of our methods on the synthetic two-spiral data and on three real data sets of MNIST, MIRFLICKR, and CIFAR, where we see that our proposed methods, with the same set of hyperparameters, can correctly adjust the network complexity to the task complexity. Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In this paper, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of our approach lies in our parameterization, where the depth, or additional complexity, is encapsulated continuously in the parameter space through control parameters that add additional complexity. We propose two methods. In tunnel networks, this selection is done at the level of a hidden unit, and in budding perceptrons, this is done at the level of a network layer; updating this control parameter introduces either another hidden unit or layer. We show the effectiveness of our methods on the synthetic two-spiral data and on three real data sets of MNIST, MIRFLICKR, and CIFAR, where we see that our proposed methods, with the same set of hyperparameters, can correctly adjust the network complexity to the task complexity.Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In this paper, we propose two novel approaches that automatically update the network structure while also learning its weights. The novelty of our approach lies in our parameterization, where the depth, or additional complexity, is encapsulated continuously in the parameter space through control parameters that add additional complexity. We propose two methods. In tunnel networks, this selection is done at the level of a hidden unit, and in budding perceptrons, this is done at the level of a network layer; updating this control parameter introduces either another hidden unit or layer. We show the effectiveness of our methods on the synthetic two-spiral data and on three real data sets of MNIST, MIRFLICKR, and CIFAR, where we see that our proposed methods, with the same set of hyperparameters, can correctly adjust the network complexity to the task complexity. |
| Author | Alpaydin, Ethem Irsoy, Ozan |
| Author_xml | – sequence: 1 givenname: Ozan orcidid: 0000-0002-7123-8361 surname: Irsoy fullname: Irsoy, Ozan email: oirsoy@bloomberg.net organization: Bloomberg LP, New York, NY, USA – sequence: 2 givenname: Ethem orcidid: 0000-0001-7506-0321 surname: Alpaydin fullname: Alpaydin, Ethem email: alpaydin@boun.edu.tr organization: Department of Computer Engineering, Boğaziçi University, Istanbul, Turkey |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/31247565$$D View this record in MEDLINE/PubMed |
| BookMark | eNp9kEtLAzEQx4NUbK39AgpS8OJl17w2j6PUJ5R6sIK3sI8JbN3u1mRX6bc3ta2HHsxlMvD7DzO_U9SrmxoQOic4JgTrm_lsNn2NKSY6ppooSpMjNKBE0IgypXp_f_neRyPvFzg8gRPB9QnqM0K5TEQyQPGkqduy7prOV-txaHzrurwtv2B8B7Aaz6BzaRVK-924D3-Gjm1aeRjt6hC9PdzPJ0_R9OXxeXI7jXKWkDYSKSssV5m12nKihSXUUqsJYYoXBETCKM6Bg-BCgdJCqIxSEFlipZJUFmyIrrdzV6757MC3Zln6HKoqrSGsasK5WITjCAvo1QG6aDpXh-1MMCGxxInUgbrcUV22hMKsXLlM3drsTQRAbYHcNd47sCYv27Qtgx6XlpUh2Gy8m1_vZuPd7LyHKD2I7qf_G7rYhkoA-AsoyTlXiv0A6NeLiQ |
| CODEN | ITNNAL |
| CitedBy_id | crossref_primary_10_1016_j_neunet_2021_08_034 crossref_primary_10_3390_a14050158 crossref_primary_10_1109_ACCESS_2024_3376525 crossref_primary_10_1145_3696003 crossref_primary_10_1016_j_bspc_2022_104185 crossref_primary_10_1177_01423312211055569 crossref_primary_10_1016_j_engappai_2022_105734 crossref_primary_10_4316_AECE_2020_02005 crossref_primary_10_3390_s22155786 |
| Cites_doi | 10.1109/72.279181 10.1109/TNN.2002.804317 10.1109/CVPR.2016.308 10.1109/ICCV.2017.298 10.1561/2200000006 10.1109/72.838999 10.1016/0893-6080(94)90058-2 10.1162/089976601317098565 10.1007/978-3-030-01246-5_1 10.1016/j.neucom.2003.10.009 10.1145/2733373.2806216 10.1109/CVPR.2016.90 10.1080/09540098908915647 10.1109/72.248452 10.1109/TNN.2004.836241 10.1109/ICMLA.2017.0-160 10.1142/S0218001409007132 10.1162/neco.1997.9.8.1735 10.1109/72.572092 10.1109/ICPR.2014.616 10.1109/72.774273 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020 |
| DBID | 97E RIA RIE AAYXX CITATION NPM 7QF 7QO 7QP 7QQ 7QR 7SC 7SE 7SP 7SR 7TA 7TB 7TK 7U5 8BQ 8FD F28 FR3 H8D JG9 JQ2 KR7 L7M L~C L~D P64 7X8 |
| DOI | 10.1109/TNNLS.2019.2918225 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef PubMed Aluminium Industry Abstracts Biotechnology Research Abstracts Calcium & Calcified Tissue Abstracts Ceramic Abstracts Chemoreception Abstracts Computer and Information Systems Abstracts Corrosion Abstracts Electronics & Communications Abstracts Engineered Materials Abstracts Materials Business File Mechanical & Transportation Engineering Abstracts Neurosciences Abstracts Solid State and Superconductivity Abstracts METADEX Technology Research Database ANTE: Abstracts in New Technology & Engineering Engineering Research Database Aerospace Database Materials Research Database ProQuest Computer Science Collection Civil Engineering Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional Biotechnology and BioEngineering Abstracts MEDLINE - Academic |
| DatabaseTitle | CrossRef PubMed Materials Research Database Technology Research Database Computer and Information Systems Abstracts – Academic Mechanical & Transportation Engineering Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Materials Business File Aerospace Database Engineered Materials Abstracts Biotechnology Research Abstracts Chemoreception Abstracts Advanced Technologies Database with Aerospace ANTE: Abstracts in New Technology & Engineering Civil Engineering Abstracts Aluminium Industry Abstracts Electronics & Communications Abstracts Ceramic Abstracts Neurosciences Abstracts METADEX Biotechnology and BioEngineering Abstracts Computer and Information Systems Abstracts Professional Solid State and Superconductivity Abstracts Engineering Research Database Calcium & Calcified Tissue Abstracts Corrosion Abstracts MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic PubMed Materials Research Database |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher – sequence: 3 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 2162-2388 |
| EndPage | 1133 |
| ExternalDocumentID | 31247565 10_1109_TNNLS_2019_2918225 8744488 |
| Genre | orig-research Journal Article |
| GroupedDBID | 0R~ 4.4 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACIWK ACPRK AENEX AFRAH AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF M43 MS~ O9- OCL PQQKQ RIA RIE RNS AAYXX CITATION NPM RIG 7QF 7QO 7QP 7QQ 7QR 7SC 7SE 7SP 7SR 7TA 7TB 7TK 7U5 8BQ 8FD F28 FR3 H8D JG9 JQ2 KR7 L7M L~C L~D P64 7X8 |
| ID | FETCH-LOGICAL-c351t-6a3df48bff9f4196f12f2f911384d1e65320ce4e6468e89668b22e6b5f78727d3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 13 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000525351800005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2162-237X 2162-2388 |
| IngestDate | Thu Oct 02 09:54:57 EDT 2025 Mon Jun 30 03:52:13 EDT 2025 Thu Jan 02 22:59:03 EST 2025 Tue Nov 18 21:49:20 EST 2025 Sat Nov 29 01:40:03 EST 2025 Wed Aug 27 02:42:21 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Issue | 4 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c351t-6a3df48bff9f4196f12f2f911384d1e65320ce4e6468e89668b22e6b5f78727d3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 |
| ORCID | 0000-0001-7506-0321 0000-0002-7123-8361 |
| PMID | 31247565 |
| PQID | 2387070579 |
| PQPubID | 85436 |
| PageCount | 10 |
| ParticipantIDs | proquest_journals_2387070579 ieee_primary_8744488 proquest_miscellaneous_2250623713 pubmed_primary_31247565 crossref_citationtrail_10_1109_TNNLS_2019_2918225 crossref_primary_10_1109_TNNLS_2019_2918225 |
| PublicationCentury | 2000 |
| PublicationDate | 2020-04-01 |
| PublicationDateYYYYMMDD | 2020-04-01 |
| PublicationDate_xml | – month: 04 year: 2020 text: 2020-04-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States – name: Piscataway |
| PublicationTitle | IEEE transaction on neural networks and learning systems |
| PublicationTitleAbbrev | TNNLS |
| PublicationTitleAlternate | IEEE Trans Neural Netw Learn Syst |
| PublicationYear | 2020 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref12 ref37 ref2 ref1 ref17 ref16 ref19 ref18 huiskes (ref34) 2008 glorot (ref32) 2011; 15 bergstra (ref13) 2012; 13 ref24 ref23 ref25 ref20 ref22 ref21 cortes (ref10) 2016 ref28 hinton (ref33) 2012 ref27 srivastava (ref11) 2015 krizhevsky (ref38) 2012 ref29 zoph (ref9) 2016 kingma (ref31) 2014 józefowicz (ref8) 2015 ref7 ref4 ref3 ref6 srivastava (ref35) 2014; 15 fahlman (ref15) 1990 lang (ref30) 1988 krizhevsky (ref36) 2009; 1 snoek (ref14) 2015 chung (ref5) 2014 huang (ref26) 2017 |
| References_xml | – ident: ref3 doi: 10.1109/72.279181 – ident: ref24 doi: 10.1109/TNN.2002.804317 – volume: 15 start-page: 315 year: 2011 ident: ref32 article-title: Deep sparse rectifier networks publication-title: J Mach Learn Res – volume: 13 start-page: 281 year: 2012 ident: ref13 article-title: Random search for hyper-parameter optimization publication-title: J Mach Learn Res – ident: ref37 doi: 10.1109/CVPR.2016.308 – ident: ref25 doi: 10.1109/ICCV.2017.298 – start-page: 1097 year: 2012 ident: ref38 article-title: ImageNet classification with deep convolutional neural networks publication-title: Proc Adv Neural Inf Process Syst – ident: ref1 doi: 10.1561/2200000006 – ident: ref20 doi: 10.1109/72.838999 – start-page: 39 year: 2008 ident: ref34 article-title: The MIR Flickr retrieval evaluation publication-title: Proc Int Conf Multimedia Inf Retr – year: 2014 ident: ref31 article-title: Adam: A method for stochastic optimization publication-title: arXiv 1412 6980 – year: 2014 ident: ref5 article-title: Empirical evaluation of gated recurrent neural networks on sequence modeling publication-title: arXiv 1412 3555 – start-page: 52 year: 1988 ident: ref30 article-title: Learning to tell two spirals apart publication-title: Proc Connectionist Models Summer School – ident: ref21 doi: 10.1016/0893-6080(94)90058-2 – volume: 1 start-page: 7 year: 2009 ident: ref36 article-title: Learning multiple layers of features from tiny images – ident: ref16 doi: 10.1162/089976601317098565 – year: 2015 ident: ref11 article-title: Highway networks publication-title: arXiv 1505 00387 – ident: ref28 doi: 10.1007/978-3-030-01246-5_1 – ident: ref19 doi: 10.1016/j.neucom.2003.10.009 – ident: ref2 doi: 10.1145/2733373.2806216 – ident: ref29 doi: 10.1109/CVPR.2016.90 – ident: ref6 doi: 10.1080/09540098908915647 – start-page: 2342 year: 2015 ident: ref8 article-title: An empirical exploration of recurrent network architectures publication-title: Proc 32nd Int Conf Mach Learn (ICML) – year: 2012 ident: ref33 article-title: Improving neural networks by preventing co-adaptation of feature detectors publication-title: arXiv 1207 0580 – ident: ref7 doi: 10.1109/72.248452 – ident: ref22 doi: 10.1109/TNN.2004.836241 – volume: 15 start-page: 2949 year: 2014 ident: ref35 article-title: Multimodal learning with deep Boltzmann machines publication-title: J Mach Learn Res – ident: ref27 doi: 10.1109/ICMLA.2017.0-160 – ident: ref23 doi: 10.1142/S0218001409007132 – year: 2016 ident: ref10 article-title: AdaNet: Adaptive structural learning of artificial neural networks publication-title: arXiv 1607 01097 – ident: ref4 doi: 10.1162/neco.1997.9.8.1735 – start-page: 2171 year: 2015 ident: ref14 article-title: Scalable Bayesian optimization using deep neural networks publication-title: Proc Int Conf Mach Learn – ident: ref17 doi: 10.1109/72.572092 – ident: ref12 doi: 10.1109/ICPR.2014.616 – start-page: 524 year: 1990 ident: ref15 article-title: The cascade-correlation learning architecture publication-title: Proc Adv Neural Inf Process Syst – ident: ref18 doi: 10.1109/72.774273 – year: 2017 ident: ref26 article-title: Data-driven sparse structure selection for deep neural networks publication-title: arXiv 1707 01213 – year: 2016 ident: ref9 article-title: Neural architecture search with reinforcement learning publication-title: arXiv 1611 01578 |
| SSID | ssj0000605649 |
| Score | 2.414827 |
| Snippet | Traditionally, deep learning algorithms update the network weights, whereas the network architecture is chosen manually using a process of trial and error. In... |
| SourceID | proquest pubmed crossref ieee |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 1124 |
| SubjectTerms | Algorithms Artificial neural networks Complexity theory Computer architecture Constructive learning Deep learning Learning algorithms Machine learning Network architecture Neural networks Parameterization Parameters Road transportation Task complexity Training |
| Title | Continuously Constructive Deep Neural Networks |
| URI | https://ieeexplore.ieee.org/document/8744488 https://www.ncbi.nlm.nih.gov/pubmed/31247565 https://www.proquest.com/docview/2387070579 https://www.proquest.com/docview/2250623713 |
| Volume | 31 |
| WOSCitedRecordID | wos000525351800005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2162-2388 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0000605649 issn: 2162-237X databaseCode: RIE dateStart: 20120101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8QwEB7cxYMX34_qulTwpl3bNM3jKOriYSmCK-yt9DEBYemKq4L_3knaLR5U8FIKTdIwmUm-SSbzAZxXiuU64SLQLOcB1_QoFJKXEvKCHBIToXGsJROZpmo20w9rcNndhUFEF3yGI_vqzvKrRflut8qubKp2Urge9KQUzV2tbj8lJFwuHNplkWABi-VsdUcm1FfTNJ082kAuPWKaILVlxv62Djlild8xpltrxlv_6-U2bLaY0r9ulGAH1rDeha0VX4Pfmu8ejGwqquf6nZz9-advqTqb5LEf6N8ivvg2UQe1kzaR4ct9eBrfTW_ug5YvISjjJHoLRB5XhqvCGG04WZaJmGGGZrNY8SpCYTkgSuQouFCoyM9RBWMoisSQ1TJZxQfQrxc1HoGfa2mqOBYh0vJflFHOVKWEMLaGKMvCg2glvaxsk4lbTot55pyKUGdO4pmVeNZK3IOLrs5Lk0rjz9J7VrRdyVaqHgxWg5S11rbMCHZImroSqT046z6TndjDj7xGEmtGTYYE9cgn9-CwGdyu7ZhAjiRke_zzP09gg1kv28XrDKBPY4OnsF5-vD0vX4ekjDM1dMr4BY902FM |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1ZS8QwEB68QF-8j3pW8E27tmmaJo_igeJaBFfYt9BjAgtSxVXBf-8k7RYfVPClFJqk5Usm_SaZzAdwVEmWq4SLQLGcB1zRpZBIXkrIC3JITITGqZb00yyTw6G6n4KT7iwMIrrgM-zZW7eXXz2X73ap7NSmaqcBNw2zCecsbE5rdSsqITFz4fguiwQLWJwOJ6dkQnU6yLL-gw3lUj2miFRbbexvfyInrfI7y3R_m6ul_33nMiy2rNI_a4bBCkxhvQpLE8UGvzXgNejZZFSj-p3c_adP34p1NuljP9C_QHzxbaoOaidrYsPH6_B4dTk4vw5axYSgjJPoLRB5XBkuC2OU4WRbJmKGGZrPYsmrCIVVgSiRo-BCoiRPRxaMoSgSQ3bL0iregJn6ucYt8HOVmiqORYhEAIoyypmspBDG1hBlWXgQTdDTZZtO3KpaPGnnVoRKO8S1RVy3iHtw3NV5aZJp_Fl6zULblWxR9WB30km6tbexJuKR0uSVpMqDw-4xWYrd_shrJFg1NRkS2SOv3IPNpnO7tmOiOSlx2-2f33kA89eDu77u32S3O7DArM_tond2YYb6Cfdgrvx4G41f992Q_ALffNqy |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Continuously+Constructive+Deep+Neural+Networks&rft.jtitle=IEEE+transaction+on+neural+networks+and+learning+systems&rft.au=Irsoy%2C+Ozan&rft.au=Alpaydin%2C+Ethem&rft.date=2020-04-01&rft.issn=2162-237X&rft.eissn=2162-2388&rft.volume=31&rft.issue=4&rft.spage=1124&rft.epage=1133&rft_id=info:doi/10.1109%2FTNNLS.2019.2918225&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TNNLS_2019_2918225 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2162-237X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2162-237X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2162-237X&client=summon |