A Stochastic Computational Multi-Layer Perceptron with Backward Propagation
Stochastic computation has recently been proposed for implementing artificial neural networks with reduced hardware and power consumption, but at a decreased accuracy and processing speed. Most existing implementations are based on pre-training such that the weights are predetermined for neurons at...
Uloženo v:
| Vydáno v: | IEEE transactions on computers Ročník 67; číslo 9; s. 1273 - 1286 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
New York
IEEE
01.09.2018
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Témata: | |
| ISSN: | 0018-9340, 1557-9956 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Stochastic computation has recently been proposed for implementing artificial neural networks with reduced hardware and power consumption, but at a decreased accuracy and processing speed. Most existing implementations are based on pre-training such that the weights are predetermined for neurons at different layers, thus these implementations lack the ability to update the values of the network parameters. In this paper, a stochastic computational multi-layer perceptron (SC-MLP) is proposed by implementing the backward propagation algorithm for updating the layer weights. Using extended stochastic logic (ESL), a reconfigurable stochastic computational activation unit (SCAU) is designed to implement different types of activation functions such as the tanh and the rectifier function. A triple modular redundancy (TMR) technique is employed for reducing the random fluctuations in stochastic computation. A probability estimator (PE) and a divider based on the TMR and a binary search algorithm are further proposed with progressive precision for reducing the required stochastic sequence length. Therefore, the latency and energy consumption of the SC-MLP are significantly reduced. The simulation results show that the proposed design is capable of implementing both the training and inference processes. For the classification of nonlinearly separable patterns, at a slight loss of accuracy by 1.32-1.34 percent, the proposed design requires only 28.5-30.1 percent of the area and 18.9-23.9 percent of the energy consumption incurred by a design using floating point arithmetic. Compared to a fixed-point implementation, the SC-MLP consumes a smaller area (40.7-45.5 percent) and a lower energy consumption (38.0-51.0 percent) with a similar processing speed and a slight drop of accuracy by 0.15-0.33 percent. The area and the energy consumption of the proposed design is from 80.7-87.1 percent and from 71.9-93.1 percent, respectively, of a binarized neural network (BNN), with a similar accuracy. |
|---|---|
| AbstractList | Stochastic computation has recently been proposed for implementing artificial neural networks with reduced hardware and power consumption, but at a decreased accuracy and processing speed. Most existing implementations are based on pre-training such that the weights are predetermined for neurons at different layers, thus these implementations lack the ability to update the values of the network parameters. In this paper, a stochastic computational multi-layer perceptron (SC-MLP) is proposed by implementing the backward propagation algorithm for updating the layer weights. Using extended stochastic logic (ESL), a reconfigurable stochastic computational activation unit (SCAU) is designed to implement different types of activation functions such as the tanh and the rectifier function. A triple modular redundancy (TMR) technique is employed for reducing the random fluctuations in stochastic computation. A probability estimator (PE) and a divider based on the TMR and a binary search algorithm are further proposed with progressive precision for reducing the required stochastic sequence length. Therefore, the latency and energy consumption of the SC-MLP are significantly reduced. The simulation results show that the proposed design is capable of implementing both the training and inference processes. For the classification of nonlinearly separable patterns, at a slight loss of accuracy by 1.32-1.34 percent, the proposed design requires only 28.5-30.1 percent of the area and 18.9-23.9 percent of the energy consumption incurred by a design using floating point arithmetic. Compared to a fixed-point implementation, the SC-MLP consumes a smaller area (40.7-45.5 percent) and a lower energy consumption (38.0-51.0 percent) with a similar processing speed and a slight drop of accuracy by 0.15-0.33 percent. The area and the energy consumption of the proposed design is from 80.7-87.1 percent and from 71.9-93.1 percent, respectively, of a binarized neural network (BNN), with a similar accuracy. |
| Author | Yanzhi Wang Jie Han Siting Liu Lombardi, Fabrizio Yidong Liu |
| Author_xml | – sequence: 1 surname: Yidong Liu fullname: Yidong Liu email: yidong1@ualberta.ca organization: Dept. of Electr. & Comput. Eng., Univ. of Alberta, Edmonton, AB, Canada – sequence: 2 surname: Siting Liu fullname: Siting Liu email: siting2@ualberta.ca organization: Dept. of Electr. & Comput. Eng., Univ. of Alberta, Edmonton, AB, Canada – sequence: 3 surname: Yanzhi Wang fullname: Yanzhi Wang email: ywang393@syr.edu organization: Electr. Eng. & Comput. Sci. Dept., Syracuse Univ., Syracuse, NY, USA – sequence: 4 givenname: Fabrizio surname: Lombardi fullname: Lombardi, Fabrizio email: lombardi@ece.neu.edu organization: Dept. of Electr. & Comput. Eng., Northeastern Univ., Boston, MA, USA – sequence: 5 surname: Jie Han fullname: Jie Han email: jhan8@ualberta.ca organization: Dept. of Electr. & Comput. Eng., Univ. of Alberta, Edmonton, AB, Canada |
| BookMark | eNp9kD1PwzAQhi1UJNrCzMASiTnt2U7ieCwRX6KISpQ5chyHuqRxsB1V_fekBDEwoBtuuPe5Oz0TNGpMoxC6xDDDGPh8nc0I4HRGUswIZSdojOOYhZzHyQiNoR-FnEZwhibObQEgIcDH6GkRvHojN8J5LYPM7NrOC69NI-rguau9DpfioGywUlaq1lvTBHvtN8GNkB97YctgZU0r3r-Rc3Raidqpi58-RW93t-vsIVy-3D9mi2UoScp9yFkKZV-MM04Yq2haxhwkgGKVSmKqiliVMlEyKooSFI5ARBQKXpWEYpxgOkXXw97Wms9OOZ9vTWf7j11OMGaYkP5Cn5oPKWmNc1ZVeWv1TthDjiE_GsvXWX40lv8Y64n4DyH1IMNboet_uKuB00qp3yspxb17Sr8Ajdh5VQ |
| CODEN | ITCOB4 |
| CitedBy_id | crossref_primary_10_1080_1448837X_2023_2206598 crossref_primary_10_1109_ACCESS_2023_3329984 crossref_primary_10_1038_s41467_024_55220_y crossref_primary_10_3390_electronics14091845 crossref_primary_10_1007_s00521_021_06617_z crossref_primary_10_1007_s13389_022_00299_6 crossref_primary_10_1109_JETCAS_2023_3244183 crossref_primary_10_1109_TCAD_2018_2858363 crossref_primary_10_1088_1361_6528_ade243 crossref_primary_10_1038_s41598_023_34146_3 crossref_primary_10_1016_j_rineng_2025_104979 crossref_primary_10_3390_app10248968 crossref_primary_10_1109_TC_2021_3056992 crossref_primary_10_3390_electronics10232985 crossref_primary_10_1016_j_adhoc_2024_103539 crossref_primary_10_1016_j_asoc_2023_110166 crossref_primary_10_1007_s10489_021_02362_x crossref_primary_10_1109_JETCAS_2018_2852705 crossref_primary_10_1109_JIOT_2023_3339623 crossref_primary_10_1007_s10586_025_05237_9 crossref_primary_10_1016_j_engappai_2019_09_003 crossref_primary_10_1109_TCSI_2023_3245022 crossref_primary_10_1109_JIOT_2025_3563942 crossref_primary_10_1109_TC_2022_3186628 crossref_primary_10_1109_TNNLS_2020_3009047 crossref_primary_10_48084_etasr_10499 crossref_primary_10_1109_ACCESS_2022_3140646 crossref_primary_10_1109_JIOT_2020_3004469 crossref_primary_10_1109_MCAS_2020_3005483 crossref_primary_10_1109_JETCAS_2023_3243950 crossref_primary_10_1007_s10922_024_09878_w crossref_primary_10_1049_iet_sen_2018_5046 crossref_primary_10_1109_TVLSI_2019_2920152 crossref_primary_10_1109_TC_2018_2885044 crossref_primary_10_1109_TC_2020_2991177 crossref_primary_10_1109_JIOT_2020_3007130 crossref_primary_10_1109_TCAD_2019_2897631 crossref_primary_10_1007_s40722_023_00282_1 crossref_primary_10_1109_TCYB_2021_3124235 crossref_primary_10_1049_ell2_12045 crossref_primary_10_1109_TCSI_2021_3103926 crossref_primary_10_1109_ACCESS_2019_2946298 crossref_primary_10_1109_TCSI_2022_3168286 crossref_primary_10_1109_ACCESS_2021_3059482 crossref_primary_10_3390_sym16121701 crossref_primary_10_1109_ACCESS_2022_3212791 crossref_primary_10_1109_ACCESS_2019_2915335 crossref_primary_10_1109_TNNLS_2023_3265533 crossref_primary_10_3390_sym11101308 crossref_primary_10_1016_j_buildenv_2021_108243 crossref_primary_10_1109_MDAT_2021_3063356 crossref_primary_10_1109_JETCAS_2023_3328875 crossref_primary_10_1109_MNANO_2022_3208757 crossref_primary_10_1007_s10586_020_03183_2 crossref_primary_10_1109_TNANO_2024_3373499 |
| Cites_doi | 10.1109/TVLSI.2017.2654298 10.1145/3061639.3062258 10.7873/DATE.2015.0377 10.1109/12.954505 10.1109/ISCAS.2009.5118213 10.1007/978-1-4899-5841-9_2 10.1109/ICASSP.2013.6638947 10.1145/2897937.2898011 10.1145/3037697.3037746 10.1109/TC.2012.231 10.1145/2744769.2747932 10.1109/TNNLS.2015.2413754 10.1109/MWSCAS.2015.7282118 10.1007/BF00194907 10.1145/3020078.3021744 10.1109/TNN.2003.816058 10.1109/12.954506 10.1145/2742060.2743758 10.1109/5.726791 10.1109/TC.2010.202 10.1109/CISS.2015.7086904 10.1109/TVLSI.2016.2535313 10.1016/j.microrel.2015.11.017 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TC.2018.2817237 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science |
| EISSN | 1557-9956 |
| EndPage | 1286 |
| ExternalDocumentID | 10_1109_TC_2018_2817237 8319953 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Natural Sciences and Engineering Research Council (NSERC) of Canada grantid: RES0025211 |
| GroupedDBID | --Z -DZ -~X .DC 0R~ 29I 4.4 5GY 6IK 85S 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AETEA AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD HZ~ IEDLZ IFIPE IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 TWZ UHB UPT XZL YZZ AAYXX ABUFD CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c289t-9780d0d07979277f38d590c00e7fe653eb5edc6ec4bbd0e140a430b9fd2311613 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 73 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000441420700005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0018-9340 |
| IngestDate | Sun Jun 29 13:14:45 EDT 2025 Tue Nov 18 20:57:53 EST 2025 Sat Nov 29 01:35:40 EST 2025 Wed Aug 27 06:00:43 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 9 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c289t-9780d0d07979277f38d590c00e7fe653eb5edc6ec4bbd0e140a430b9fd2311613 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0003-0004-3804 0000-0002-8849-4994 0000-0003-0505-8183 0000-0003-3152-3245 |
| PQID | 2117122780 |
| PQPubID | 85452 |
| PageCount | 14 |
| ParticipantIDs | crossref_primary_10_1109_TC_2018_2817237 ieee_primary_8319953 crossref_citationtrail_10_1109_TC_2018_2817237 proquest_journals_2117122780 |
| PublicationCentury | 2000 |
| PublicationDate | 2018-09-01 |
| PublicationDateYYYYMMDD | 2018-09-01 |
| PublicationDate_xml | – month: 09 year: 2018 text: 2018-09-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on computers |
| PublicationTitleAbbrev | TC |
| PublicationYear | 2018 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref14 ioffe (ref18) 2015 ref11 hubara (ref19) 2016 ref10 ref2 ref17 glorot (ref16) 2011 courbariaux (ref20) 2015 netzer (ref15) 2011 haykin (ref1) 2009 ref24 ref23 ref26 ref25 ref22 ref21 ref28 ref27 ref29 ref8 ref7 ref9 ref4 ref3 ref6 ref5 |
| References_xml | – ident: ref10 doi: 10.1109/TVLSI.2017.2654298 – ident: ref7 doi: 10.1145/3061639.3062258 – ident: ref8 doi: 10.7873/DATE.2015.0377 – ident: ref5 doi: 10.1109/12.954505 – ident: ref28 doi: 10.1109/ISCAS.2009.5118213 – ident: ref4 doi: 10.1007/978-1-4899-5841-9_2 – ident: ref17 doi: 10.1109/ICASSP.2013.6638947 – ident: ref12 doi: 10.1145/2897937.2898011 – ident: ref13 doi: 10.1145/3037697.3037746 – ident: ref25 doi: 10.1109/TC.2012.231 – ident: ref29 doi: 10.1145/2744769.2747932 – ident: ref9 doi: 10.1109/TNNLS.2015.2413754 – ident: ref24 doi: 10.1109/MWSCAS.2015.7282118 – start-page: 315 year: 2011 ident: ref16 article-title: Deep sparse rectifier neural networks publication-title: Proc 14th Int Conf Artif Intell Statist – ident: ref2 doi: 10.1007/BF00194907 – ident: ref21 doi: 10.1145/3020078.3021744 – ident: ref3 doi: 10.1109/TNN.2003.816058 – ident: ref6 doi: 10.1109/12.954506 – ident: ref26 doi: 10.1145/2742060.2743758 – start-page: 3123 year: 2015 ident: ref20 article-title: BinaryConnect: Training deep neural networks with binary weights during propagations publication-title: Proc 28th Int Conf Neural Inf Process Syst – ident: ref14 doi: 10.1109/5.726791 – start-page: 4107 year: 2016 ident: ref19 article-title: Binarized neural networks publication-title: Proc 30th Conf Neural Inf Process Syst – year: 2011 ident: ref15 article-title: Reading digits in natural images with unsupervised feature learning publication-title: Proc NIPS Workshop on Deep Learning and Unsupervised Feature Learning – ident: ref23 doi: 10.1109/TC.2010.202 – ident: ref11 doi: 10.1109/CISS.2015.7086904 – start-page: 448 year: 2015 ident: ref18 article-title: Batch normalization: Accelerating deep network training by reducing internal covariate shift publication-title: Proc Int Conf Mach Learn – year: 2009 ident: ref1 publication-title: Neural Networks and Learning Machines – ident: ref27 doi: 10.1109/TVLSI.2016.2535313 – ident: ref22 doi: 10.1016/j.microrel.2015.11.017 |
| SSID | ssj0006209 |
| Score | 2.5045512 |
| Snippet | Stochastic computation has recently been proposed for implementing artificial neural networks with reduced hardware and power consumption, but at a decreased... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 1273 |
| SubjectTerms | Accuracy Activation Artificial neural networks Backpropagation binary search Biological neural networks Computation Computer simulation Energy consumption Fixed point arithmetic Floating point arithmetic Hardware Mathematical analysis multi-layer perceptron Multilayer perceptrons neural network Neural networks Neurons Power consumption probability estimator Propagation Rectifiers Redundancy Search algorithms Stochastic computation Training Tunneling magnetoresistance Variation |
| Title | A Stochastic Computational Multi-Layer Perceptron with Backward Propagation |
| URI | https://ieeexplore.ieee.org/document/8319953 https://www.proquest.com/docview/2117122780 |
| Volume | 67 |
| WOSCitedRecordID | wos000441420700005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1557-9956 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0006209 issn: 0018-9340 databaseCode: RIE dateStart: 19680101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEJ4A8aAHUdCIounBgwcXCt3dtkckEhMNIRETbpvtS00MGFj8_bbdQkjUg9nLHmY2m06n8-jMNwDXOsVxzmRuNc2wKNbG6pwgcWRckoE4xLFE-GETdDxmsxmfVOB22wujtfbFZ7rjXv1dvlrItUuVdRlxDcWkClVK07JXa3vqpptyjp5VYBLjAOPTw7w7HboSLtbpM2ut3cDzHQvkR6r8OIe9cRnV__dbR3AYnEg0KKV-DBU9b0B9M6ABBX1twMEO2mATHgfouVjIt9xBM6OSOqQCke_DjZ5y64CjSVnrslzMkcvSojuX4rMbCU2WNsJ-9Swn8DK6nw4fojBLIZI2pCoiBzSk7EM55X1KDWEq4VhirKnRaUK0SLSSqZaxEAprG3blMcGCG2UdQOsVklOozRdzfQbIaJmkVPGkn8jYMMUlE0YwkRvcM4qmLehs1jeTAWjczbv4yHzAgXk2HWZOIFkQSAtutgyfJcbG36RNt_5bsrD0LWhvBJgFHVxlNrSlPdfpi89_57qAffftsmKsDbViudaXsCe_ivfV8spvr28ioM0A |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB60CurBt1ifOXjw4Na02d0kRy2KYi0FK3hbNi8VpJXa-vvNZNMiqAfZyx4y7JLJJDOTme8DOLE5TUuhS29pTiSpdd7mFEsTh0kGhohjmQpkE7zbFU9PsjcHZ7NeGGttKD6zDXwNd_lmqCeYKjsXDBuK2TwsIHNW7Naa7bv5tKCj6U2YpTQC-TSpPO-3sYhLNFrCn9dIef7tDAqkKj924nC8XK_978fWYTW6keSi0vsGzNnBJqxNKRpItNhNWPmGN7gFdxfkYTzULyWCM5NqdEwGktCJm3RK74KTXlXtMhoOCOZpySUm-fxSIr2Rj7Gfg8g2PF5f9ds3SWRTSLQPqsYJQg0Z_3DJZYtzx4TJJNWUWu5snjGrMmt0bnWqlKHWB15lyqiSzngX0PuFbAdqg-HA7gJxVmc5NzJrZTp1wkgtlFNClY42neF5HRrT-S10hBpHxou3IoQcVBb9doEKKaJC6nA6E3ivUDb-HrqF8z8bFqe-DgdTBRbRCj8KH9zyJvb60r3fpY5h6aZ_3yk6t927fVjG71T1YwdQG48m9hAW9ef49WN0FJbaF_ne0Ek |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Stochastic+Computational+Multi-Layer+Perceptron+with+Backward+Propagation&rft.jtitle=IEEE+transactions+on+computers&rft.au=Liu%2C+Yidong&rft.au=Liu%2C+Siting&rft.au=Wang%2C+Yanzhi&rft.au=Lombardi%2C+Fabrizio&rft.date=2018-09-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=0018-9340&rft.eissn=1557-9956&rft.volume=67&rft.issue=9&rft.spage=1273&rft_id=info:doi/10.1109%2FTC.2018.2817237&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9340&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9340&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9340&client=summon |