Convergence analysis of the batch gradient-based neuro-fuzzy learning algorithm with smoothing L1/2 regularization for the first-order Takagi–Sugeno system
It has been proven that Takagi–Sugeno systems are universal approximators, and they are applied widely to classification and regression problems. The main challenges of these models are convergence analysis and their computational complexity due to the large number of connections and the pruning of...
Uloženo v:
| Vydáno v: | Fuzzy sets and systems Ročník 319; s. 28 - 49 |
|---|---|
| Hlavní autoři: | , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
Elsevier B.V
15.07.2017
|
| Témata: | |
| ISSN: | 0165-0114, 1872-6801 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | It has been proven that Takagi–Sugeno systems are universal approximators, and they are applied widely to classification and regression problems. The main challenges of these models are convergence analysis and their computational complexity due to the large number of connections and the pruning of unnecessary parameters. The neuro-fuzzy learning algorithm involves two tasks: generating comparable sparse networks and training the parameters. In addition, regularization methods have attracted increasing attention for network pruning, particularly the Lq(0<q<1) regularizer after L1 regularization, which can obtain better solutions to sparsity problems. The L1/2 regularizer has a specific sparsity capacity and it is representative of Lq(0<q<1) regularizations. However, the nonsmoothness of the L1/2 regularizer may lead to oscillations in the learning process. In this study, we propose a gradient-based neuro-fuzzy learning algorithm with a smoothing L1/2 regularization for the first-order Takagi–Sugeno fuzzy inference system. The proposed approach has the following three advantages: (i) it enhances the original L1/2 regularizer by eliminating the oscillation of the gradient in the cost function during the training; (ii) it performs better by pruning inactive connections, where the number of the redundant connections for removal is higher than that generated by the original L1/2 regularizer, while it is also implemented by simultaneous structure and parameter learning processes; and (iii) it is possible to demonstrate the theoretical convergence analysis of this learning method, which we focus on explicitly. We also provide a series of simulations to demonstrate that the smoothing L1/2 regularization can often obtain more compressive representations than the current L1/2 regularization. |
|---|---|
| AbstractList | It has been proven that Takagi–Sugeno systems are universal approximators, and they are applied widely to classification and regression problems. The main challenges of these models are convergence analysis and their computational complexity due to the large number of connections and the pruning of unnecessary parameters. The neuro-fuzzy learning algorithm involves two tasks: generating comparable sparse networks and training the parameters. In addition, regularization methods have attracted increasing attention for network pruning, particularly the Lq(0<q<1) regularizer after L1 regularization, which can obtain better solutions to sparsity problems. The L1/2 regularizer has a specific sparsity capacity and it is representative of Lq(0<q<1) regularizations. However, the nonsmoothness of the L1/2 regularizer may lead to oscillations in the learning process. In this study, we propose a gradient-based neuro-fuzzy learning algorithm with a smoothing L1/2 regularization for the first-order Takagi–Sugeno fuzzy inference system. The proposed approach has the following three advantages: (i) it enhances the original L1/2 regularizer by eliminating the oscillation of the gradient in the cost function during the training; (ii) it performs better by pruning inactive connections, where the number of the redundant connections for removal is higher than that generated by the original L1/2 regularizer, while it is also implemented by simultaneous structure and parameter learning processes; and (iii) it is possible to demonstrate the theoretical convergence analysis of this learning method, which we focus on explicitly. We also provide a series of simulations to demonstrate that the smoothing L1/2 regularization can often obtain more compressive representations than the current L1/2 regularization. |
| Author | Liu, Yan Yang, Dakun |
| Author_xml | – sequence: 1 givenname: Yan surname: Liu fullname: Liu, Yan email: liuyan@dlpu.edu.cn organization: School of Information Science and Engineering, Dalian Polytechnic University, Dalian 116034, China – sequence: 2 givenname: Dakun surname: Yang fullname: Yang, Dakun organization: School of Information Science and Technology, Sun Yat-sen University, Guangzhou 510006, China |
| BookMark | eNotkE1OwzAQhS1UJNrCAdj5AkntOM2PWKGKP6kSC8racpxx4pLaku0WtSvuwJrLcRIcYDPzpDfzZvTN0MRYAwhdU5JSQovFNlXep1mUKSlTQtgZmtKqzJKiInSCptFYJoTS_ALNvN8SEnVBpuhrZc0BXAdGAhZGDEevPbYKhx5wI4LscedEq8GEpBEeWmxg72yi9qfTEQ8gnNGmw2LorNOh3-H3WLHfWRv60VjTRYYddPtBOH0SQVuDlXW_8Uo7HxLrWnB4I95Ep78_Pl_28ReL_dEH2F2icyUGD1f_fY5e7-82q8dk_fzwtLpdJ0CzLCSVbCRrKgE5K-uqUDWVIIAsFahcMlLTvKYlLRjJmGobqNs6L-KGJFA14yybo5u_XIhHDhoc91KPSFrtQAbeWs0p4SNpvuWRNB9Jc1LySJr9AMNdesU |
| ContentType | Journal Article |
| Copyright | 2016 Elsevier B.V. |
| Copyright_xml | – notice: 2016 Elsevier B.V. |
| DOI | 10.1016/j.fss.2016.07.003 |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Mathematics |
| EISSN | 1872-6801 |
| EndPage | 49 |
| ExternalDocumentID | S0165011416302287 |
| GrantInformation_xml | – fundername: Natural Science Foundation Guidance Project of Liaoning Province grantid: 201602050 – fundername: China Postdoctoral Science Foundation grantid: 2015M581334 funderid: http://dx.doi.org/10.13039/501100002858 – fundername: National Natural Science Foundation of China grantid: 61403056 funderid: http://dx.doi.org/10.13039/501100001809 |
| GroupedDBID | --K --M -~X .DC .~1 0R~ 1B1 1RT 1~. 1~5 4.4 457 4G. 5VS 7-5 71M 8P~ 9JN 9JO AAAKF AABNK AACTN AAEDT AAEDW AAIAV AAIKJ AAKOC AALRI AAOAW AAQFI AARIN AAXUO AAYFN ABAOU ABBOA ABFNM ABJNI ABMAC ABUCO ABYKQ ACAZW ACDAQ ACGFS ACNCT ACRLP ACZNC ADBBV ADEZE ADGUI ADTZH AEBSH AECPX AEKER AENEX AFKWA AFTJW AGHFR AGUBO AGYEJ AHHHB AHJVU AHZHX AIALX AIEXJ AIGVJ AIKHN AITUG AJBFU AJOXV ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AOUOD APLSM ARUGR AXJTR BJAXD BKOJK BLXMC CS3 DU5 EBS EFJIC EFLBG EJD EO8 EO9 EP2 EP3 FDB FIRID FNPLU FYGXN G-Q GBLVA GBOLZ HAMUX IHE J1W JJJVA K-O KOM LG9 LY1 M41 MHUIS MO0 N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. Q38 RIG ROL RPZ SDF SDG SDP SDS SES SPC SPCBC SSB SSD SST SSV SSW SSZ T5K TN5 WH7 ZMT ~02 ~G- |
| ID | FETCH-LOGICAL-e122t-8cbc3b8ae437986f91ceae05fef4c30914917163023fdbe9d946bc3c0e8b91ce3 |
| ISSN | 0165-0114 |
| IngestDate | Fri Feb 23 02:36:05 EST 2024 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | First-order Takagi–Sugeno inference system Pi–Sigma network Convergence Smoothing L1/2 regularization |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-e122t-8cbc3b8ae437986f91ceae05fef4c30914917163023fdbe9d946bc3c0e8b91ce3 |
| PageCount | 22 |
| ParticipantIDs | elsevier_sciencedirect_doi_10_1016_j_fss_2016_07_003 |
| PublicationCentury | 2000 |
| PublicationDate | 2017-07-15 |
| PublicationDateYYYYMMDD | 2017-07-15 |
| PublicationDate_xml | – month: 07 year: 2017 text: 2017-07-15 day: 15 |
| PublicationDecade | 2010 |
| PublicationTitle | Fuzzy sets and systems |
| PublicationYear | 2017 |
| Publisher | Elsevier B.V |
| Publisher_xml | – name: Elsevier B.V |
| References | Quah, Quek (br0050) 2006; 36 Wu, Li, Yang, Liu (br0300) 2010; 180 Davanipoor, Zekri, Sheikholeslam (br0130) 2012; 20 Sun, Au, Choi (br0240) 2007; 37 Turkmen, Guney (br0020) 2008; 35 Husaini (br0090) 2012; 9 Chakraborty, Pal (br0230) 2004; 15 Wang, Er, Meng (br0120) 2010; 20 Pedrycz (br0010) 1984; 13 Kukolj, Levi (br0290) 2004; 34 Yu, Li (br0260) 2004; 12 del Campo, Echanobe, Bosque, Tarela (br0250) 2008; 16 Jin (br0150) 2000; 8 Mackay (br0320) 1992; 4 Caydas, Hascalik, Ekici (br0310) 2009; 36 Yu, Li, Luo, Su, Li (br0100) 2010; 31 Pal, Saha (br0140) 2008; 38 Ghazalia, Hussainb, Nawia, Mohamada (br0080) 2009; 72 Kim, Kim, Lee (br0160) 2006; 36 Kisi, Haktanir, Ardiclioglu, Ozturk, Yalcin, Uludag (br0030) 2009; 40 Xu (br0210) 2010 Shin, Ghosh (br0070) 1991 Angelov, Filev (br0060) 2004; 34 Han, Wu, Qiao (br0110) 2014; 44 Saito, Nakano (br0200) 2000; 12 McLoone, Irwin (br0180) 2001; 37 Chang, Xu, Zhang, Wang, Liang (br0190) 2010; 40 Sugeno, Takagi (br0280) 1983; 9 Song, Wang, Tu, Marmarelis, Hampson, Deadwyler, Berger (br0170) 2013; 35 Takagi, Sugeno (br0040) 1985; 15 Stoeva, Nikov (br0270) 2000; 112 Xu, Zhang, Wang, Chang (br0220) 2009; 52 |
| References_xml | – volume: 4 start-page: 415 year: 1992 end-page: 447 ident: br0320 article-title: Bayesian interpolation publication-title: Neural Comput. – volume: 52 start-page: 1 year: 2009 end-page: 9 ident: br0220 article-title: regularizer publication-title: Sci. China, Ser. F, Inf. Sci. – volume: 20 start-page: 463 year: 2012 end-page: 470 ident: br0130 article-title: Fuzzy wavelet neural network with an accelerated hybrid learning algorithm publication-title: IEEE Trans. Fuzzy Syst. – volume: 16 start-page: 761 year: 2008 end-page: 778 ident: br0250 article-title: Efficient hardware/software implementation of an adaptive neuro-fuzzy system publication-title: IEEE Trans. Fuzzy Syst. – volume: 112 start-page: 27 year: 2000 end-page: 39 ident: br0270 article-title: A fuzzy backpropagation algorithm publication-title: Fuzzy Sets Syst. – volume: 72 start-page: 2359 year: 2009 end-page: 2367 ident: br0080 article-title: Non-stationary and stationary prediction of financial time series using dynamic ridge polynomial neural network publication-title: Neurocomputing – start-page: 3151 year: 2010 end-page: 3184 ident: br0210 article-title: Data modeling: visual psychology approach and L1/2 regularization theory publication-title: Proceedings of the International Congress of Mathematicians – start-page: 13 year: 1991 end-page: 18 ident: br0070 article-title: The Pi–Sigma networks: an efficient higher-order neural network for pattern classification and function approximation publication-title: Proceedings of International Joint Conference on Neural Networks, vol. 1 – volume: 9 start-page: 313 year: 1983 end-page: 325 ident: br0280 article-title: Multi-dimensional fuzzy reasoning publication-title: Fuzzy Sets Syst. – volume: 12 start-page: 709 year: 2000 end-page: 729 ident: br0200 article-title: Second-order learning algorithm with squared penalty term publication-title: Neural Comput. – volume: 15 start-page: 110 year: 2004 end-page: 123 ident: br0230 article-title: A neuro-fuzzy scheme for simultaneous feature selection and fuzzy rule-based classification publication-title: IEEE Trans. Neural Netw. – volume: 36 start-page: 1006 year: 2006 end-page: 1023 ident: br0160 article-title: Evolving compact and interpretable Takagi–Sugeno fuzzy models with a new encoding scheme publication-title: IEEE Trans. Syst. Man Cybern., Part B, Cybern. – volume: 180 start-page: 1630 year: 2010 end-page: 1642 ident: br0300 article-title: A modified gradient-based neuro-fuzzy learning algorithm and its convergence publication-title: Inf. Sci. – volume: 36 start-page: 6135 year: 2009 end-page: 6139 ident: br0310 article-title: An adaptive neuro-fuzzy inference system (ANFIS) model for wire–EDM publication-title: Expert Syst. Appl. – volume: 37 start-page: 71 year: 2001 end-page: 90 ident: br0180 article-title: Improving neural network training solutions using regularisation publication-title: Neurocomputing – volume: 8 start-page: 212 year: 2000 end-page: 221 ident: br0150 article-title: Fuzzy modeling of high-dimensional systems: complexity reduction and interpretability improvement publication-title: IEEE Trans. Fuzzy Syst. – volume: 35 start-page: 1657 year: 2008 end-page: 1667 ident: br0020 article-title: Genetic tracker with adaptive neuro-fuzzy inference system for multiple target tracking publication-title: Expert Syst. Appl. – volume: 44 start-page: 554 year: 2014 end-page: 564 ident: br0110 article-title: Nonlinear systems modeling based on self-organizing fuzzy-neural-network with adaptive computation algorithm publication-title: IEEE Trans. Cybern. – volume: 36 start-page: 166 year: 2006 end-page: 178 ident: br0050 article-title: FITSK: online local learning with generic fuzzy input Takagi–Sugeno–Kang fuzzy framework for nonlinear system estimation publication-title: IEEE Trans. Syst. Man Cybern., Part B, Cybern. – volume: 20 start-page: 389 year: 2010 end-page: 403 ident: br0120 article-title: An online self-organizing scheme for parsimonious and accurate fuzzy neural networks publication-title: Int. J. Neural Syst. – volume: 35 start-page: 335 year: 2013 end-page: 357 ident: br0170 article-title: Identification of sparse neural functional connectivity using penalized likelihood estimation and basis functions publication-title: J. Comput. Neurosci. – volume: 12 start-page: 411 year: 2004 end-page: 420 ident: br0260 article-title: Fuzzy identification using fuzzy neural networks with stable learning algorithms publication-title: IEEE Trans. Fuzzy Syst. – volume: 15 start-page: 116 year: 1985 end-page: 132 ident: br0040 article-title: Fuzzy identification of systems and its applications to modeling and control publication-title: IEEE Trans. Syst. Man Cybern. – volume: 9 start-page: 440 year: 2012 end-page: 447 ident: br0090 article-title: The effect of network parameters on Pi–Sigma neural network for temperature forecasting publication-title: Int. J. Mod. Phys. Conf. Ser. – volume: 31 start-page: 3282 year: 2010 end-page: 3288 ident: br0100 article-title: Prediction of the mechanical properties of the post-forged Ti–6Al–4V alloy using fuzzy neural network publication-title: Mater. Des. – volume: 13 start-page: 153 year: 1984 end-page: 167 ident: br0010 article-title: An identification algorithm in fuzzy relational systems publication-title: Fuzzy Sets Syst. – volume: 34 start-page: 484 year: 2004 end-page: 498 ident: br0060 article-title: An approach to online identification of Takagi–Sugeno fuzzy models publication-title: IEEE Trans. Syst. Man Cybern., Part B, Cybern. – volume: 38 start-page: 1626 year: 2008 end-page: 1638 ident: br0140 article-title: Simultaneous structure identification and fuzzy rule generation for Takagi–Sugeno models publication-title: IEEE Trans. Syst. Man Cybern., Part B, Cybern. – volume: 40 start-page: 438 year: 2009 end-page: 444 ident: br0030 article-title: Adaptive neuro-fuzzy computing technique for suspended sediment estimation publication-title: Adv. Eng. Softw. – volume: 34 start-page: 272 year: 2004 end-page: 282 ident: br0290 article-title: Identification of complex systems based on neural and Takagi–Sugeno fuzzy model publication-title: IEEE Trans. Syst. Man Cybern., Part B, Cybern. – volume: 40 start-page: 985 year: 2010 end-page: 998 ident: br0190 article-title: Robust regularization theory based on publication-title: Sci. China – volume: 37 start-page: 1321 year: 2007 end-page: 1331 ident: br0240 article-title: A neuro-fuzzy inference system through integration of fuzzy logic and extreme learning machines publication-title: IEEE Trans. Syst. Man Cybern., Part B, Cybern. |
| SSID | ssj0001160 |
| Score | 2.2951643 |
| Snippet | It has been proven that Takagi–Sugeno systems are universal approximators, and they are applied widely to classification and regression problems. The main... |
| SourceID | elsevier |
| SourceType | Publisher |
| StartPage | 28 |
| SubjectTerms | Convergence First-order Takagi–Sugeno inference system Pi–Sigma network Smoothing [formula omitted] regularization |
| Title | Convergence analysis of the batch gradient-based neuro-fuzzy learning algorithm with smoothing L1/2 regularization for the first-order Takagi–Sugeno system |
| URI | https://dx.doi.org/10.1016/j.fss.2016.07.003 |
| Volume | 319 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVESC databaseName: Elsevier SD Freedom Collection Journals 2021 customDbUrl: eissn: 1872-6801 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001160 issn: 0165-0114 databaseCode: AIEXJ dateStart: 19950113 isFulltext: true titleUrlDefault: https://www.sciencedirect.com providerName: Elsevier |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV3JbtRAEG0NCQc4IFaFsKgP3KwO4yV29zEKiQBBhJRBGk6Wl-6Js3giL1GUE__AEfFzfAlV3W1Ph1wAiYtlWV5arqeqcvnVK0JeAYQUjnpjZRIWLCoTnwkkUxUZzzkKVKntTA-bSA4O-HwuPk0m34demIvTpK755aU4_6-mhmNgbGyd_QtzjzeFA7APRoctmB22f2T4XeSRN0ZjM3MkRzDDzMHxHnmLRvO8OoYhrPS0pCVT_dXVOERi4WWni2VTdUdnlsR-tgST6o4o3d3hNXqGfWO7OEeyoqogm2Raz9ObZSfZohrYFOFhj3qwVjrazYn39ZNb2Rm56NbRUEemUNXrMLFC8Rdb4n6TnfS1W7XwdTnU9G2aUtqNdhpT3YyRSWi6Srek8cg8CVjMbcXDuuzQulnrdLkTvo0A6o3AYGoUx1uqRY123yi2TsNVFBy5iYe4ClwEZKooDpTcIutBsi3A66_vvNubvx8Dve_rJvRx1cNPc00f_O1BTq7j5C-z--Se_fCgOwYwD8hE1g_J3Y-jam_7iPxwoEMH6NClonAO1dCh16FDHejQATp0hA5F6NAROvSD_zqg14FDATj69g5wqAHOz6_fDGSoQcRj8nl_b7b7ltnxHUz6QdAxXuRFmPNMouQlj5XwC5lJ8A1SRUUIeWokUKsJp1apMpeiFFEMVxRTyXM8N3xC1uplLTcIlUrg2IZymucqmspYJD58iEDur-IgErF8SqLh7aY2czQZYQr2Twci43EKRknRKOkUWRfh5r9d9ozcWUH6OVnrml6-ILeLi65qm5cWJb8A8iyZJw |
| linkProvider | Elsevier |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Convergence+analysis+of+the+batch+gradient-based+neuro-fuzzy+learning+algorithm+with+smoothing+L1%2F2+regularization+for+the+first-order+Takagi%E2%80%93Sugeno+system&rft.jtitle=Fuzzy+sets+and+systems&rft.au=Liu%2C+Yan&rft.au=Yang%2C+Dakun&rft.date=2017-07-15&rft.pub=Elsevier+B.V&rft.issn=0165-0114&rft.eissn=1872-6801&rft.volume=319&rft.spage=28&rft.epage=49&rft_id=info:doi/10.1016%2Fj.fss.2016.07.003&rft.externalDocID=S0165011416302287 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0165-0114&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0165-0114&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0165-0114&client=summon |