A Fully Probabilistic Model for Sigmoid Approximation and Its Hardware- Efficient Implementation
The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential and reciprocal operations. Existing studies applied piecewise models to approximate sigmoid function and employed numerical methods or non-u...
Saved in:
| Published in: | IEEE transactions on circuits and systems. I, Regular papers Vol. 71; no. 8; pp. 3775 - 3786 |
|---|---|
| Main Authors: | , , , , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
IEEE
01.08.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 1549-8328, 1558-0806 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential and reciprocal operations. Existing studies applied piecewise models to approximate sigmoid function and employed numerical methods or non-uniform input segmentations to mitigate fitting inaccuracies. However, the breakpoints introduce inevitable approximation precision loss. Besides, additional fitting processes greatly increase hardware complexity and power consumption. This paper presents a hardware-friendly sigmoidal approximation from the perspective of probability theory. We find that for a given input, the output of a sigmoid function can be approximated by the probability that the sum of this input and a Gaussian random variable is greater than or equal to zero. As the derived theorem does not involve piecewise expressions, the precision loss caused by the breakpoint issue is avoided. A low-complexity binary-search-based address localization method is proposed to optimize our theorem for hardware implementation. For the optimized scheme, an efficient implemented circuit is also presented. Our scheme's approximation ability and hardware efficiency are validated through software modeling and FPGA- and ASIC-based experiments. Feedforward neural network-based classification applications demonstrate that building networks with the proposed sigmoid approximator has only a tiny recognition rate loss. |
|---|---|
| AbstractList | The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential and reciprocal operations. Existing studies applied piecewise models to approximate sigmoid function and employed numerical methods or non-uniform input segmentations to mitigate fitting inaccuracies. However, the breakpoints introduce inevitable approximation precision loss. Besides, additional fitting processes greatly increase hardware complexity and power consumption. This paper presents a hardware-friendly sigmoidal approximation from the perspective of probability theory. We find that for a given input, the output of a sigmoid function can be approximated by the probability that the sum of this input and a Gaussian random variable is greater than or equal to zero. As the derived theorem does not involve piecewise expressions, the precision loss caused by the breakpoint issue is avoided. A low-complexity binary-search-based address localization method is proposed to optimize our theorem for hardware implementation. For the optimized scheme, an efficient implemented circuit is also presented. Our scheme’s approximation ability and hardware efficiency are validated through software modeling and FPGA- and ASIC-based experiments. Feedforward neural network-based classification applications demonstrate that building networks with the proposed sigmoid approximator has only a tiny recognition rate loss. |
| Author | Lu, Wenhao Lu, Zhongzhiguang Zhang, Xiangfen Lu, Minshan Sun, Miao Dong, Boyi Shu, Zhou |
| Author_xml | – sequence: 1 givenname: Wenhao orcidid: 0000-0002-4842-2400 surname: Lu fullname: Lu, Wenhao email: wenhaolu3-c@my.cityu.edu.hk organization: Department of Electrical Engineering, City University of Hong Kong, Kowloon Tong, Hong Kong – sequence: 2 givenname: Minshan surname: Lu fullname: Lu, Minshan email: minshan001@e.ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 3 givenname: Xiangfen orcidid: 0000-0002-5332-3175 surname: Zhang fullname: Zhang, Xiangfen email: xiangfen@shnu.edu.cn organization: College of Information, Mechanical and Electrical Engineering, Shanghai Normal University, Shanghai, China – sequence: 4 givenname: Zhongzhiguang surname: Lu fullname: Lu, Zhongzhiguang email: luzh0013@e.ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 5 givenname: Miao orcidid: 0000-0002-4537-6998 surname: Sun fullname: Sun, Miao email: miao.sun@ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 6 givenname: Boyi surname: Dong fullname: Dong, Boyi email: boyi.dong@ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 7 givenname: Zhou orcidid: 0000-0002-8493-4526 surname: Shu fullname: Shu, Zhou email: shuzhou@nus.edu.sg organization: Department of Electrical and Computer Engineering, National University of Singapore, Cluny Road, Singapore |
| BookMark | eNpNkM1OwzAQhC1UJNrCAyBxsMQ5xf9xjlXV0khFILWcjRs7yFUSBzsV9O1JaA-cdg7f7O7MBIwa31gA7jGaYYyyp91im88IImxGGaY8RVdgjDmXCZJIjAbNskRSIm_AJMYDQiRDFI_BxxyujlV1gm_B7_XeVS52roAv3tgKlj7ArfusvTNw3rbB_7had843UDcG5l2Eax3Mtw42gcuydIWzTQfzuq1s3as_9BZcl7qK9u4yp-B9tdwt1snm9TlfzDdJQZjoEsIKRiXTTBcZMpIQSQuc7gXnQxpuCCkRSYnUUmiRCUM5EwVDmRHG2pJSOgWP5739m19HGzt18MfQ9CcVRVIwSlPJegqfqSL4GIMtVRv6TOGkMFJDkWooUg1FqkuRvefh7HHW2n88FzLLBP0FKAZv8w |
| CODEN | ITCSCH |
| Cites_doi | 10.1109/ASAP.2019.00018 10.1109/ICCIT.2008.131 10.1109/ISIT.2019.8849497 10.1109/ICFPT51103.2020.00014 10.1109/TCSI.2022.3209574 10.1109/AICAS.2019.8771531 10.3390/bdcc2030026 10.1109/TIE.2022.3146573 10.1109/72.977323 10.1109/JBHI.2022.3178660 10.1109/TCSI.2019.2939563 10.1049/ip-cds:19971587 10.1109/TCSI.2021.3102303 10.1109/TCSI.2021.3133931 10.3390/electronics11091365 10.1109/ATS49688.2020.9301543 10.1109/TCC.2020.2997008 10.1007/springerreference_184570 10.1016/j.jvcir.2019.06.007 10.1109/TCSII.2020.2999458 10.1109/TCSI.2019.2959886 10.1109/TCSI.2022.3231863 10.1109/12.537127 10.1109/ICENCO48310.2019.9027479 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024 |
| DBID | 97E RIA RIE AAYXX CITATION 7SP 8FD L7M |
| DOI | 10.1109/TCSI.2024.3413570 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Electronics & Communications Abstracts Technology Research Database Advanced Technologies Database with Aerospace |
| DatabaseTitle | CrossRef Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1558-0806 |
| EndPage | 3786 |
| ExternalDocumentID | 10_1109_TCSI_2024_3413570 10568996 |
| Genre | orig-research |
| GroupedDBID | 0R~ 29I 4.4 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACIWK AETIX AGQYO AGSQL AHBIQ AIBXA AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD HZ~ H~9 IFIPE IPLJI JAVBF M43 O9- OCL PZZ RIA RIE RNS VJK AAYXX CITATION 7SP 8FD L7M |
| ID | FETCH-LOGICAL-c246t-24c4384a4ac90d82283c17b65513575d22f02728a86a696d3546c409d6deef333 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001258745300001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1549-8328 |
| IngestDate | Mon Jun 30 10:21:05 EDT 2025 Sat Nov 29 06:24:00 EST 2025 Wed Aug 27 02:34:34 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 8 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c246t-24c4384a4ac90d82283c17b65513575d22f02728a86a696d3546c409d6deef333 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0002-5332-3175 0000-0002-4842-2400 0000-0002-4537-6998 0000-0002-8493-4526 |
| PQID | 3086433784 |
| PQPubID | 85411 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_10568996 proquest_journals_3086433784 crossref_primary_10_1109_TCSI_2024_3413570 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-08-01 |
| PublicationDateYYYYMMDD | 2024-08-01 |
| PublicationDate_xml | – month: 08 year: 2024 text: 2024-08-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on circuits and systems. I, Regular papers |
| PublicationTitleAbbrev | TCSI |
| PublicationYear | 2024 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref15 ref14 ref11 ref10 ref2 ref1 ref17 ref16 ref18 ref24 ref23 ref22 ref21 ref28 Knuth (ref20) 1973; 422 ref27 ref8 Haley (ref19) 1953 ref7 Xiao (ref26) 2017 ref9 ref4 ref3 ref6 ref5 LeCun (ref25) 2010 |
| References_xml | – ident: ref14 doi: 10.1109/ASAP.2019.00018 – ident: ref23 doi: 10.1109/ICCIT.2008.131 – ident: ref9 doi: 10.1109/ISIT.2019.8849497 – ident: ref22 doi: 10.1109/ICFPT51103.2020.00014 – year: 2017 ident: ref26 article-title: Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms publication-title: arXiv:1708.07747 – ident: ref2 doi: 10.1109/TCSI.2022.3209574 – ident: ref3 doi: 10.1109/AICAS.2019.8771531 – ident: ref6 doi: 10.3390/bdcc2030026 – ident: ref13 doi: 10.1109/TIE.2022.3146573 – ident: ref27 doi: 10.1109/72.977323 – ident: ref10 doi: 10.1109/JBHI.2022.3178660 – ident: ref17 doi: 10.1109/TCSI.2019.2939563 – ident: ref11 doi: 10.1049/ip-cds:19971587 – volume-title: MNIST handwritten digit database year: 2010 ident: ref25 – ident: ref5 doi: 10.1109/TCSI.2021.3102303 – volume: 422 start-page: 559 volume-title: The Art of Computer Programming: Sorting and Searching year: 1973 ident: ref20 article-title: Sorting and searching – ident: ref18 doi: 10.1109/TCSI.2021.3133931 – volume-title: Estimation Dosage Mortality Relationship When Dose is Subject to Error year: 1953 ident: ref19 – ident: ref16 doi: 10.3390/electronics11091365 – ident: ref28 doi: 10.1109/ATS49688.2020.9301543 – ident: ref8 doi: 10.1109/TCC.2020.2997008 – ident: ref24 doi: 10.1007/springerreference_184570 – ident: ref7 doi: 10.1016/j.jvcir.2019.06.007 – ident: ref15 doi: 10.1109/TCSII.2020.2999458 – ident: ref4 doi: 10.1109/TCSI.2019.2959886 – ident: ref1 doi: 10.1109/TCSI.2022.3231863 – ident: ref12 doi: 10.1109/12.537127 – ident: ref21 doi: 10.1109/ICENCO48310.2019.9027479 |
| SSID | ssj0029031 |
| Score | 2.4227586 |
| Snippet | The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Index Database Publisher |
| StartPage | 3775 |
| SubjectTerms | Approximation Artificial neural networks ASIC implementation Complexity Computational modeling FPGA implementation Hardware hardware efficiency Integrated circuit modeling Localization method Neural networks Numerical methods Numerical models Optimization Probabilistic logic Probabilistic models probabilistic-based method Probability theory Random variables sigmoid activation function Statistical analysis Theorems |
| Title | A Fully Probabilistic Model for Sigmoid Approximation and Its Hardware- Efficient Implementation |
| URI | https://ieeexplore.ieee.org/document/10568996 https://www.proquest.com/docview/3086433784 |
| Volume | 71 |
| WOSCitedRecordID | wos001258745300001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-0806 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0029031 issn: 1549-8328 databaseCode: RIE dateStart: 20040101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwEA9u-KAPfk6cTsmDT0K3NknT5nGMDQcyBpuwt5olqQxcJ13nx39vLu1kID741oc2hLvc5X69u98hdBeT1LpGoT0tjAUoihNPqDDwiOQikMzIuXQkro_RaBTPZmJcNau7XhhjjCs-M214dLl8vVIb-FXWgSnxFh_wGqpFES-btX7QlfBpSY7KhGePaVylMANfdKa9ydBCQcLa4LNDGEy8cwm5qSq_XLG7XwbH_9zZCTqqAkncLTV_ivZMdoYOd-gFz9FzFwPC_MLj3FotVMECKTOG8Wev2AareLJ4Wa4WGneBWPxzUXYxYplpPCzWGJL6HzI3Hu47ngm7A-zIhJdVv1LWQE-D_rT34FUTFTxFGC88whSjMZNMKuHrGKhvVBDNOUx5sXGbJiS1MJXEMuZWV1zTkHFlEaDm2piUUnqB6tkqM5cICz_U1tZlOJfWCaSRCJWynlVHIvWjVKdNdL8VcfJWEmckDnD4IgF9JKCPpNJHEzVApjsvluJsotZWK0llW-uEWhTGKI1idvXHZ9foAFYv6_RaqF7kG3OD9tV7sVjnt-7YfAOQKr9Y |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED90CuqDnxOnU_Pgk1DtkjRtHodsbDiH4IS91SxJZaCdbPPrvzeXVhmID771oSXhLne5X-_udwBnCc2ca5QmMNI6gKIFDaSOGgFVQjYUt2qkPIlrL-73k-FQ3pbN6r4Xxlrri8_sBT76XL6Z6Ff8VXaJU-IdPhDLsBJxTsOiXesHX8mQFfSoXAbuoCZlErMRysvB1V3XgUHKL9BrRziaeOEa8nNVfjljf8O0t_65t23YLENJ0ix0vwNLNt-FjQWCwT14aBLEmJ_kdursFutgkZaZ4AC0J-LCVXI3fnyejA1pIrX4x7joYyQqN6Q7nxFM67-rqQ1IyzNNuB0QTyf8XHYs5VW4b7cGV52gnKkQaMrFPKBcc5ZwxZWWoUmQ_EY34pHAOS8ucjOUZg6o0kQlwmlLGBZxoR0GNMJYmzHG9qGST3J7AESGkXHWrqKRcm4gi2WktfOtJpZZGGcmq8H5t4jTl4I6I_WQI5Qp6iNFfaSlPmpQRZkuvFiIswb1b62kpXXNUuZwGGcsTvjhH5-dwlpncNNLe93-9RGs40pF1V4dKvPpqz2GVf02H8-mJ_4IfQGxW8Kf |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Fully+Probabilistic+Model+for+Sigmoid+Approximation+and+Its+Hardware-+Efficient+Implementation&rft.jtitle=IEEE+transactions+on+circuits+and+systems.+I%2C+Regular+papers&rft.au=Lu%2C+Wenhao&rft.au=Lu%2C+Minshan&rft.au=Zhang%2C+Xiangfen&rft.au=Lu%2C+Zhongzhiguang&rft.date=2024-08-01&rft.pub=IEEE&rft.issn=1549-8328&rft.volume=71&rft.issue=8&rft.spage=3775&rft.epage=3786&rft_id=info:doi/10.1109%2FTCSI.2024.3413570&rft.externalDocID=10568996 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1549-8328&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1549-8328&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1549-8328&client=summon |