A Fully Probabilistic Model for Sigmoid Approximation and Its Hardware- Efficient Implementation

The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential and reciprocal operations. Existing studies applied piecewise models to approximate sigmoid function and employed numerical methods or non-u...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	IEEE transactions on circuits and systems. I, Regular papers Ročník 71; číslo 8; s. 3775 - 3786
Hlavní autoři:	Lu, Wenhao, Lu, Minshan, Zhang, Xiangfen, Lu, Zhongzhiguang, Sun, Miao, Dong, Boyi, Shu, Zhou
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	New York IEEE 01.08.2024 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:	Approximation Artificial neural networks ASIC implementation Complexity Computational modeling FPGA implementation Hardware hardware efficiency Integrated circuit modeling Localization method Neural networks Numerical methods Numerical models Optimization Probabilistic logic Probabilistic models probabilistic-based method Probability theory Random variables sigmoid activation function Statistical analysis Theorems
ISSN:	1549-8328, 1558-0806
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Abstract	The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential and reciprocal operations. Existing studies applied piecewise models to approximate sigmoid function and employed numerical methods or non-uniform input segmentations to mitigate fitting inaccuracies. However, the breakpoints introduce inevitable approximation precision loss. Besides, additional fitting processes greatly increase hardware complexity and power consumption. This paper presents a hardware-friendly sigmoidal approximation from the perspective of probability theory. We find that for a given input, the output of a sigmoid function can be approximated by the probability that the sum of this input and a Gaussian random variable is greater than or equal to zero. As the derived theorem does not involve piecewise expressions, the precision loss caused by the breakpoint issue is avoided. A low-complexity binary-search-based address localization method is proposed to optimize our theorem for hardware implementation. For the optimized scheme, an efficient implemented circuit is also presented. Our scheme's approximation ability and hardware efficiency are validated through software modeling and FPGA- and ASIC-based experiments. Feedforward neural network-based classification applications demonstrate that building networks with the proposed sigmoid approximator has only a tiny recognition rate loss.
AbstractList	The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential and reciprocal operations. Existing studies applied piecewise models to approximate sigmoid function and employed numerical methods or non-uniform input segmentations to mitigate fitting inaccuracies. However, the breakpoints introduce inevitable approximation precision loss. Besides, additional fitting processes greatly increase hardware complexity and power consumption. This paper presents a hardware-friendly sigmoidal approximation from the perspective of probability theory. We find that for a given input, the output of a sigmoid function can be approximated by the probability that the sum of this input and a Gaussian random variable is greater than or equal to zero. As the derived theorem does not involve piecewise expressions, the precision loss caused by the breakpoint issue is avoided. A low-complexity binary-search-based address localization method is proposed to optimize our theorem for hardware implementation. For the optimized scheme, an efficient implemented circuit is also presented. Our scheme’s approximation ability and hardware efficiency are validated through software modeling and FPGA- and ASIC-based experiments. Feedforward neural network-based classification applications demonstrate that building networks with the proposed sigmoid approximator has only a tiny recognition rate loss.
Author	Lu, Wenhao Lu, Zhongzhiguang Zhang, Xiangfen Lu, Minshan Sun, Miao Dong, Boyi Shu, Zhou
Author_xml	– sequence: 1 givenname: Wenhao orcidid: 0000-0002-4842-2400 surname: Lu fullname: Lu, Wenhao email: wenhaolu3-c@my.cityu.edu.hk organization: Department of Electrical Engineering, City University of Hong Kong, Kowloon Tong, Hong Kong – sequence: 2 givenname: Minshan surname: Lu fullname: Lu, Minshan email: minshan001@e.ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 3 givenname: Xiangfen orcidid: 0000-0002-5332-3175 surname: Zhang fullname: Zhang, Xiangfen email: xiangfen@shnu.edu.cn organization: College of Information, Mechanical and Electrical Engineering, Shanghai Normal University, Shanghai, China – sequence: 4 givenname: Zhongzhiguang surname: Lu fullname: Lu, Zhongzhiguang email: luzh0013@e.ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 5 givenname: Miao orcidid: 0000-0002-4537-6998 surname: Sun fullname: Sun, Miao email: miao.sun@ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 6 givenname: Boyi surname: Dong fullname: Dong, Boyi email: boyi.dong@ntu.edu.sg organization: School of Electrical and Electronic Engineering, Nanyang Technological University, Jurong West, Singapore – sequence: 7 givenname: Zhou orcidid: 0000-0002-8493-4526 surname: Shu fullname: Shu, Zhou email: shuzhou@nus.edu.sg organization: Department of Electrical and Computer Engineering, National University of Singapore, Cluny Road, Singapore
BookMark	eNpNkM1OwzAQhC1UJNrCAyBxsMQ5xf9xjlXV0khFILWcjRs7yFUSBzsV9O1JaA-cdg7f7O7MBIwa31gA7jGaYYyyp91im88IImxGGaY8RVdgjDmXCZJIjAbNskRSIm_AJMYDQiRDFI_BxxyujlV1gm_B7_XeVS52roAv3tgKlj7ArfusvTNw3rbB_7had843UDcG5l2Eax3Mtw42gcuydIWzTQfzuq1s3as_9BZcl7qK9u4yp-B9tdwt1snm9TlfzDdJQZjoEsIKRiXTTBcZMpIQSQuc7gXnQxpuCCkRSYnUUmiRCUM5EwVDmRHG2pJSOgWP5739m19HGzt18MfQ9CcVRVIwSlPJegqfqSL4GIMtVRv6TOGkMFJDkWooUg1FqkuRvefh7HHW2n88FzLLBP0FKAZv8w
CODEN	ITCSCH
Cites_doi	10.1109/ASAP.2019.00018 10.1109/ICCIT.2008.131 10.1109/ISIT.2019.8849497 10.1109/ICFPT51103.2020.00014 10.1109/TCSI.2022.3209574 10.1109/AICAS.2019.8771531 10.3390/bdcc2030026 10.1109/TIE.2022.3146573 10.1109/72.977323 10.1109/JBHI.2022.3178660 10.1109/TCSI.2019.2939563 10.1049/ip-cds:19971587 10.1109/TCSI.2021.3102303 10.1109/TCSI.2021.3133931 10.3390/electronics11091365 10.1109/ATS49688.2020.9301543 10.1109/TCC.2020.2997008 10.1007/springerreference_184570 10.1016/j.jvcir.2019.06.007 10.1109/TCSII.2020.2999458 10.1109/TCSI.2019.2959886 10.1109/TCSI.2022.3231863 10.1109/12.537127 10.1109/ICENCO48310.2019.9027479
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
DBID	97E RIA RIE AAYXX CITATION 7SP 8FD L7M
DOI	10.1109/TCSI.2024.3413570
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Electronics & Communications Abstracts Technology Research Database Advanced Technologies Database with Aerospace
DatabaseTitle	CrossRef Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	1558-0806
EndPage	3786
ExternalDocumentID	10_1109_TCSI_2024_3413570 10568996
Genre	orig-research
GroupedDBID	0R~ 29I 4.4 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACIWK AETIX AGQYO AGSQL AHBIQ AIBXA AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD HZ~ H~9 IFIPE IPLJI JAVBF M43 O9- OCL PZZ RIA RIE RNS VJK AAYXX CITATION 7SP 8FD L7M
ID	FETCH-LOGICAL-c246t-24c4384a4ac90d82283c17b65513575d22f02728a86a696d3546c409d6deef333
IEDL.DBID	RIE
ISICitedReferencesCount	0
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001258745300001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1549-8328
IngestDate	Mon Jun 30 10:21:05 EDT 2025 Sat Nov 29 06:24:00 EST 2025 Wed Aug 27 02:34:34 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	8
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c246t-24c4384a4ac90d82283c17b65513575d22f02728a86a696d3546c409d6deef333
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-5332-3175 0000-0002-4842-2400 0000-0002-4537-6998 0000-0002-8493-4526
PQID	3086433784
PQPubID	85411
PageCount	12
ParticipantIDs	ieee_primary_10568996 proquest_journals_3086433784 crossref_primary_10_1109_TCSI_2024_3413570
PublicationCentury	2000
PublicationDate	2024-08-01
PublicationDateYYYYMMDD	2024-08-01
PublicationDate_xml	– month: 08 year: 2024 text: 2024-08-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on circuits and systems. I, Regular papers
PublicationTitleAbbrev	TCSI
PublicationYear	2024
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref12 ref15 ref14 ref11 ref10 ref2 ref1 ref17 ref16 ref18 ref24 ref23 ref22 ref21 ref28 Knuth (ref20) 1973; 422 ref27 ref8 Haley (ref19) 1953 ref7 Xiao (ref26) 2017 ref9 ref4 ref3 ref6 ref5 LeCun (ref25) 2010
References_xml	– ident: ref14 doi: 10.1109/ASAP.2019.00018 – ident: ref23 doi: 10.1109/ICCIT.2008.131 – ident: ref9 doi: 10.1109/ISIT.2019.8849497 – ident: ref22 doi: 10.1109/ICFPT51103.2020.00014 – year: 2017 ident: ref26 article-title: Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms publication-title: arXiv:1708.07747 – ident: ref2 doi: 10.1109/TCSI.2022.3209574 – ident: ref3 doi: 10.1109/AICAS.2019.8771531 – ident: ref6 doi: 10.3390/bdcc2030026 – ident: ref13 doi: 10.1109/TIE.2022.3146573 – ident: ref27 doi: 10.1109/72.977323 – ident: ref10 doi: 10.1109/JBHI.2022.3178660 – ident: ref17 doi: 10.1109/TCSI.2019.2939563 – ident: ref11 doi: 10.1049/ip-cds:19971587 – volume-title: MNIST handwritten digit database year: 2010 ident: ref25 – ident: ref5 doi: 10.1109/TCSI.2021.3102303 – volume: 422 start-page: 559 volume-title: The Art of Computer Programming: Sorting and Searching year: 1973 ident: ref20 article-title: Sorting and searching – ident: ref18 doi: 10.1109/TCSI.2021.3133931 – volume-title: Estimation Dosage Mortality Relationship When Dose is Subject to Error year: 1953 ident: ref19 – ident: ref16 doi: 10.3390/electronics11091365 – ident: ref28 doi: 10.1109/ATS49688.2020.9301543 – ident: ref8 doi: 10.1109/TCC.2020.2997008 – ident: ref24 doi: 10.1007/springerreference_184570 – ident: ref7 doi: 10.1016/j.jvcir.2019.06.007 – ident: ref15 doi: 10.1109/TCSII.2020.2999458 – ident: ref4 doi: 10.1109/TCSI.2019.2959886 – ident: ref1 doi: 10.1109/TCSI.2022.3231863 – ident: ref12 doi: 10.1109/12.537127 – ident: ref21 doi: 10.1109/ICENCO48310.2019.9027479
SSID	ssj0029031
Score	2.4227586
Snippet	The sigmoid function is a representative activation function in shallow neural networks. Its hardware realization is challenging due to the complex exponential...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Index Database Publisher
StartPage	3775
SubjectTerms	Approximation Artificial neural networks ASIC implementation Complexity Computational modeling FPGA implementation Hardware hardware efficiency Integrated circuit modeling Localization method Neural networks Numerical methods Numerical models Optimization Probabilistic logic Probabilistic models probabilistic-based method Probability theory Random variables sigmoid activation function Statistical analysis Theorems
Title	A Fully Probabilistic Model for Sigmoid Approximation and Its Hardware- Efficient Implementation
URI	https://ieeexplore.ieee.org/document/10568996 https://www.proquest.com/docview/3086433784
Volume	71
WOSCitedRecordID	wos001258745300001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-0806 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0029031 issn: 1549-8328 databaseCode: RIE dateStart: 20040101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA4qHvTgW6wvcvAkpF032dnkWMSiIKXQKr2taZKVgt3KtvXx781kVymIB2972CxhJpnMt5n5PkIuVJJoG6WO8REIJmQimJZSM5TXtn5EDIFM5_E-7XblcKh6dbN66IVxzoXiM9fEx3CXb6dmgb_KWqgS7_EBrJLVNIWqWesHXamIV-SoQjG_TGV9hXkVqdbgun_noWAsmhizExQmXjqEgqrKr1AczpfO9j9ntkO26kSStivP75IVV-yRzSV6wX3y1KaIMD9pr_S7FqtgkZSZovzZC_XJKu2PnyfTsaVtJBb_GFddjFQXlt7NZxQv9d916Ri9CTwTfgY0kAlP6n6l4oA8dG4G17esVlRgJhYwZ7EwgkuhhTYqshKpb8xVOgJUefF5m43j3MPUWGoJGhRYnggwHgFasM7lnPNDslZMC3dEaD7SBsD5_CEHHwRAmSS3HCxE3jje7w1y-W3i7LUizsgC4IhUhv7I0B9Z7Y8GOUCbLr1YmbNBTr-9ktV7a5Zxj8IE56kUx38MOyEb-PWqTu-UrM3LhTsj6-ZtPp6V52HZfAEMFb9c
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB58gXrwWbE-c_AkRLebbJociygWaxFapbc1TbJSsFtp6-vfm8muUhAP3vawYcNMMplvM_N9ACcqSbSN6o6yvuCUy4RTLaWmKK9t_YhYBDKdh1a93Za9nrorm9VDL4xzLhSfuTN8DHf5dmRe8VfZOarEe3wg5mEx4TyOinatH3ylIlbQo3JF_UKV5SVmLVLn3YtO04PBmJ9h1E5QmnjmGAq6Kr-CcThhrtb_ObcNWCtTSdIofL8Jcy7fgtUZgsFteGwQxJif5G7s9y3WwSItM0EBtGfi01XSGTwNRwNLGkgt_jEo-hiJzi1pTicEr_Xf9dhRchmYJvwMSKATHpYdS3kF7q8uuxfXtNRUoCbmYkpjbjiTXHNtVGQlkt-YWr0vUOfFZ242jjMPVGOppdBCCcsSLozHgFZY5zLG2A4s5KPc7QLJ-toI4XwGkQkfBoQySWaZsCLyxvGer8Lpt4nTl4I6Iw2QI1Ip-iNFf6SlP6pQQZvOvFiYswoH315Jy901SZnHYZyxuuR7fww7huXr7m0rbTXbN_uwgl8qqvYOYGE6fnWHsGTepoPJ-CgsoS8s2sKj
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Fully+Probabilistic+Model+for+Sigmoid+Approximation+and+Its+Hardware-+Efficient+Implementation&rft.jtitle=IEEE+transactions+on+circuits+and+systems.+I%2C+Regular+papers&rft.au=Lu%2C+Wenhao&rft.au=Lu%2C+Minshan&rft.au=Zhang%2C+Xiangfen&rft.au=Lu%2C+Zhongzhiguang&rft.date=2024-08-01&rft.pub=IEEE&rft.issn=1549-8328&rft.volume=71&rft.issue=8&rft.spage=3775&rft.epage=3786&rft_id=info:doi/10.1109%2FTCSI.2024.3413570&rft.externalDocID=10568996
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1549-8328&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1549-8328&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1549-8328&client=summon