Improving transfer learning accuracy by reusing Stacked Denoising Autoencoders
Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study...
Saved in:
| Published in: | Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics pp. 1380 - 1387 |
|---|---|
| Main Authors: | , , , , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
01.10.2014
|
| Subjects: | |
| ISSN: | 1062-922X |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study transference on deep architectures, known to directly extract the features from the input data. A Stacked Denoising Autoencoder (SDA) is a deep model able to represent the hierarchical features needed for solving classification problems. In this paper we study the performance of SDAs trained on one problem and reused to solve a different problem not only with different distribution but also with a different tasks. We propose two different approaches: 1) unsupervised feature transference, and 2) supervised feature transference using deep transfer learning. We show that SDAs using the unsupervised feature transference outperform randomly initialized machines on a new problem. We achieved 7% relative improvement on average error rate and 41% on average computation time to classify typed uppercase letters. In the case of supervised feature transference, we achieved 5.7% relative improvement in the average error rate, by reusing the first and second hidden layer, and 8.5% relative improvement for the average error rate and 54% speed up w.r.t the baseline by reusing all three hidden layers for the same data. We also explore transfer learning between geometrical shapes and canonical shapes, we achieved 7.4% relative improvement on average error rate in case of supervised feature transference approach. |
|---|---|
| AbstractList | Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study transference on deep architectures, known to directly extract the features from the input data. A Stacked Denoising Autoencoder (SDA) is a deep model able to represent the hierarchical features needed for solving classification problems. In this paper we study the performance of SDAs trained on one problem and reused to solve a different problem not only with different distribution but also with a different tasks. We propose two different approaches: 1) unsupervised feature transference, and 2) supervised feature transference using deep transfer learning. We show that SDAs using the unsupervised feature transference outperform randomly initialized machines on a new problem. We achieved 7% relative improvement on average error rate and 41% on average computation time to classify typed uppercase letters. In the case of supervised feature transference, we achieved 5.7% relative improvement in the average error rate, by reusing the first and second hidden layer, and 8.5% relative improvement for the average error rate and 54% speed up w.r.t the baseline by reusing all three hidden layers for the same data. We also explore transfer learning between geometrical shapes and canonical shapes, we achieved 7.4% relative improvement on average error rate in case of supervised feature transference approach. |
| Author | Kandaswamy, Chetak Silva, Luis M. Sousa, Ricardo de Sa, Joaquim Marques Santos, Jorge M. Alexandre, Luis A. |
| Author_xml | – sequence: 1 givenname: Chetak surname: Kandaswamy fullname: Kandaswamy, Chetak email: chetak.kand@gmail.com organization: Inst. de Eng. Biomed. (INEB), Portugal – sequence: 2 givenname: Luis M. surname: Silva fullname: Silva, Luis M. email: lmas@ua.pt organization: INEB, Porto, Portugal – sequence: 3 givenname: Luis A. surname: Alexandre fullname: Alexandre, Luis A. organization: Univ. da Beira Interior, Covilhã, Portugal – sequence: 4 givenname: Ricardo surname: Sousa fullname: Sousa, Ricardo organization: INEB, Univ. do Porto, Porto, Portugal – sequence: 5 givenname: Jorge M. surname: Santos fullname: Santos, Jorge M. organization: INEB, Porto, Portugal – sequence: 6 givenname: Joaquim Marques surname: de Sa fullname: de Sa, Joaquim Marques organization: INEB, Porto, Portugal |
| BookMark | eNotz0FLwzAYxvEIE9zm7oKXfoHW903TJjmOqnMw9TAFbyNN3kp1S0fSCv32Mt3pgd_hgf-MTXznibEbhAwR9N32uco4oMhKLQWCvGALLRUKqXWuBMgJmyKUPNWcf1yxWYxfABwEqil7WR-Ooftp_WfSB-NjQyHZkwn-JMbaIRg7JvWYBBriyba9sd_kknvyXfsny6HvyNvOUYjX7LIx-0iL887Z--PDW_WUbl5X62q5SVuURZ82gjgq0tZpjiVZBFC2tooKJ3lJhTSaVIFcNxxdKcCZ2hneyEYi1rIw-Zzd_v-2RLQ7hvZgwrg75-e_I_hRZg |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/SMC.2014.6974107 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Sciences (General) |
| EISBN | 9781479938407 1479938408 |
| EndPage | 1387 |
| ExternalDocumentID | 6974107 |
| Genre | orig-research |
| GroupedDBID | 29F 6IE 6IF 6IH 6IK 6IL 6IM 6IN AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS |
| ID | FETCH-LOGICAL-i175t-f4e218e9cd9216ec1008cbc8e5d726e57a9e85129f21d640dabda2f7f711b75a3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 25 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000370963701084&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1062-922X |
| IngestDate | Wed Aug 27 04:46:46 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i175t-f4e218e9cd9216ec1008cbc8e5d726e57a9e85129f21d640dabda2f7f711b75a3 |
| PageCount | 8 |
| ParticipantIDs | ieee_primary_6974107 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-Oct. |
| PublicationDateYYYYMMDD | 2014-10-01 |
| PublicationDate_xml | – month: 10 year: 2014 text: 2014-Oct. |
| PublicationDecade | 2010 |
| PublicationTitle | Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics |
| PublicationTitleAbbrev | SMC |
| PublicationYear | 2014 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0020418 |
| Score | 2.0978892 |
| Snippet | Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 1380 |
| SubjectTerms | Computer architecture Deep Learning Error analysis Noise reduction Shape Training Transfer Learning Visualization Yttrium |
| Title | Improving transfer learning accuracy by reusing Stacked Denoising Autoencoders |
| URI | https://ieeexplore.ieee.org/document/6974107 |
| WOSCitedRecordID | wos000370963701084&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3LSsQwFA3j4EI3Og_xTRYuFOxMk7ZJs5TRwYUOAz6Y3ZDHjcymlU4rzN_btLUquHEXAoGQS7gnueeci9CFLzTRIgg9GwfWC2MKniSKeVGkeMCYtn5lX_z6wGezeLEQ8w66brUwAFCRz2DkhlUt36S6cF9lY1aCX-Kk41ucs1qr1T6u_JDUsjdGPUHp4qsk6Yvx0-PEcbjCUbP-VyOVKo9M9_63g300_Bbk4XmbanqoA0kf7f7wEuyjXnNL1_iysZK-GqBZ-2eA8wqhQoabPhFvWGpdZFJvsNrgDBwB_g2X2LO81gbfQpKuqpmbIk-d2aUjPA_Ry_TueXLvNR0UvFUJC3LPhlCmcBDaCEoYaOfko5WOITKcMoi4FBC7lG8pMSz0jVRGUsstJ0TxSAYHqJukCRwirCwxpnzR-kLaUAqtrFJEMhsaoamv9REauONavtcmGcvmpI7_nj5BOy4iNSvuFHXzrIAztK0_8tU6O68i-wkkyKUT |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEA2lCupF-yF-m4MHBbfdpNns5ihqqdiWglV6K_mYlF66st0V-u_dbNeq4MVbyCGEDOFNMu-9QejKF5po0WGejTrWYxEFTxLFvSBQYYdzbf3CvvitHw6H0WQiRhV0u9HCAEBBPoOWGxa1fBPrzH2VtXme_BInHd9ynbNKtdbmeeUzsha-ceoJSidfRUlftF8G947FxVrlCr9aqRRI0t3_3x4OUPNbkodHG7CpoQos6mjvh5tgHdXKe7rE16WZ9E0DDTe_BjgtclRIcNkpYoal1lki9QqrFU7AUeBnOM8-84tt8AMs4nkxc5elsbO7dJTnJnrtPo7ve17ZQ8Gb54lB6lkGOYiD0EZQwkE7Lx-tdASBCSmHIJQCIgf6lhLDmW-kMpLa0IaEqDCQnUNUXcQLOEJYWWJM_qb1hbRMCq2sUkRyy4zQ1Nf6GDXccU3f1zYZ0_KkTv6evkQ7vfGgP-0_DZ9P0a6Lzpojd4aqaZLBOdrWH-l8mVwUUf4E6-WoXA |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Conference+proceedings+-+IEEE+International+Conference+on+Systems%2C+Man%2C+and+Cybernetics&rft.atitle=Improving+transfer+learning+accuracy+by+reusing+Stacked+Denoising+Autoencoders&rft.au=Kandaswamy%2C+Chetak&rft.au=Silva%2C+Luis+M.&rft.au=Alexandre%2C+Luis+A.&rft.au=Sousa%2C+Ricardo&rft.date=2014-10-01&rft.pub=IEEE&rft.issn=1062-922X&rft.spage=1380&rft.epage=1387&rft_id=info:doi/10.1109%2FSMC.2014.6974107&rft.externalDocID=6974107 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1062-922X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1062-922X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1062-922X&client=summon |