Improving transfer learning accuracy by reusing Stacked Denoising Autoencoders
Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study...
Saved in:
| Published in: | Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics pp. 1380 - 1387 |
|---|---|
| Main Authors: | , , , , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
01.10.2014
|
| Subjects: | |
| ISSN: | 1062-922X |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study transference on deep architectures, known to directly extract the features from the input data. A Stacked Denoising Autoencoder (SDA) is a deep model able to represent the hierarchical features needed for solving classification problems. In this paper we study the performance of SDAs trained on one problem and reused to solve a different problem not only with different distribution but also with a different tasks. We propose two different approaches: 1) unsupervised feature transference, and 2) supervised feature transference using deep transfer learning. We show that SDAs using the unsupervised feature transference outperform randomly initialized machines on a new problem. We achieved 7% relative improvement on average error rate and 41% on average computation time to classify typed uppercase letters. In the case of supervised feature transference, we achieved 5.7% relative improvement in the average error rate, by reusing the first and second hidden layer, and 8.5% relative improvement for the average error rate and 54% speed up w.r.t the baseline by reusing all three hidden layers for the same data. We also explore transfer learning between geometrical shapes and canonical shapes, we achieved 7.4% relative improvement on average error rate in case of supervised feature transference approach. |
|---|---|
| AbstractList | Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow architectures show low performance as they are generally based on hand-crafted features obtained from experts. It is therefore interesting to study transference on deep architectures, known to directly extract the features from the input data. A Stacked Denoising Autoencoder (SDA) is a deep model able to represent the hierarchical features needed for solving classification problems. In this paper we study the performance of SDAs trained on one problem and reused to solve a different problem not only with different distribution but also with a different tasks. We propose two different approaches: 1) unsupervised feature transference, and 2) supervised feature transference using deep transfer learning. We show that SDAs using the unsupervised feature transference outperform randomly initialized machines on a new problem. We achieved 7% relative improvement on average error rate and 41% on average computation time to classify typed uppercase letters. In the case of supervised feature transference, we achieved 5.7% relative improvement in the average error rate, by reusing the first and second hidden layer, and 8.5% relative improvement for the average error rate and 54% speed up w.r.t the baseline by reusing all three hidden layers for the same data. We also explore transfer learning between geometrical shapes and canonical shapes, we achieved 7.4% relative improvement on average error rate in case of supervised feature transference approach. |
| Author | Kandaswamy, Chetak Silva, Luis M. Sousa, Ricardo de Sa, Joaquim Marques Santos, Jorge M. Alexandre, Luis A. |
| Author_xml | – sequence: 1 givenname: Chetak surname: Kandaswamy fullname: Kandaswamy, Chetak email: chetak.kand@gmail.com organization: Inst. de Eng. Biomed. (INEB), Portugal – sequence: 2 givenname: Luis M. surname: Silva fullname: Silva, Luis M. email: lmas@ua.pt organization: INEB, Porto, Portugal – sequence: 3 givenname: Luis A. surname: Alexandre fullname: Alexandre, Luis A. organization: Univ. da Beira Interior, Covilhã, Portugal – sequence: 4 givenname: Ricardo surname: Sousa fullname: Sousa, Ricardo organization: INEB, Univ. do Porto, Porto, Portugal – sequence: 5 givenname: Jorge M. surname: Santos fullname: Santos, Jorge M. organization: INEB, Porto, Portugal – sequence: 6 givenname: Joaquim Marques surname: de Sa fullname: de Sa, Joaquim Marques organization: INEB, Porto, Portugal |
| BookMark | eNotz0FLwzAYxvEIE9zm7oKXfoHW903TJjmOqnMw9TAFbyNN3kp1S0fSCv32Mt3pgd_hgf-MTXznibEbhAwR9N32uco4oMhKLQWCvGALLRUKqXWuBMgJmyKUPNWcf1yxWYxfABwEqil7WR-Ooftp_WfSB-NjQyHZkwn-JMbaIRg7JvWYBBriyba9sd_kknvyXfsny6HvyNvOUYjX7LIx-0iL887Z--PDW_WUbl5X62q5SVuURZ82gjgq0tZpjiVZBFC2tooKJ3lJhTSaVIFcNxxdKcCZ2hneyEYi1rIw-Zzd_v-2RLQ7hvZgwrg75-e_I_hRZg |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/SMC.2014.6974107 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Sciences (General) |
| EISBN | 9781479938407 1479938408 |
| EndPage | 1387 |
| ExternalDocumentID | 6974107 |
| Genre | orig-research |
| GroupedDBID | 29F 6IE 6IF 6IH 6IK 6IL 6IM 6IN AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS |
| ID | FETCH-LOGICAL-i175t-f4e218e9cd9216ec1008cbc8e5d726e57a9e85129f21d640dabda2f7f711b75a3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 25 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000370963701084&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1062-922X |
| IngestDate | Wed Aug 27 04:46:46 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i175t-f4e218e9cd9216ec1008cbc8e5d726e57a9e85129f21d640dabda2f7f711b75a3 |
| PageCount | 8 |
| ParticipantIDs | ieee_primary_6974107 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-Oct. |
| PublicationDateYYYYMMDD | 2014-10-01 |
| PublicationDate_xml | – month: 10 year: 2014 text: 2014-Oct. |
| PublicationDecade | 2010 |
| PublicationTitle | Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics |
| PublicationTitleAbbrev | SMC |
| PublicationYear | 2014 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0020418 |
| Score | 2.0978892 |
| Snippet | Transfer learning is a process that allows reusing a learning machine trained on a problem to solve a new problem. Transfer learning studies on shallow... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 1380 |
| SubjectTerms | Computer architecture Deep Learning Error analysis Noise reduction Shape Training Transfer Learning Visualization Yttrium |
| Title | Improving transfer learning accuracy by reusing Stacked Denoising Autoencoders |
| URI | https://ieeexplore.ieee.org/document/6974107 |
| WOSCitedRecordID | wos000370963701084&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFH_M4UEvug_xmxw8KJitzdKmOcp0eHEMVNhtpMmr7NJK1wr7703aWhW8eAuBQEh4vPeS3wfAlYcTbTufmCJ6HuXGM1TqSFEjEucw6bRxeWU2IebzaLmUiw7ctlwYRKzAZzhyw-ov32S6dE9l49AWv76jju8IEdZcrba58rhf095CRiVjy68vSU-On5-mDsPFR836X0YqVR6ZHfxvB4cw_CbkkUWbanrQwbQP-z-0BPvQa6J0Q64bKembAczbNwNSVBUq5qTxiXgjSusyV3pL4i3J0QHg34itPW1YG3KPabauZu7KInNilw7wPITX2cPL9JE2Dgp0bcuCgiYcbQpHqY1kfojaKfnoWEcYGMFCDISSGLmUnzDfhNwzKjaKJSIRvh-LQE2OoJtmKR4DUUbIyEycQlbAE8FibptqZHHCI6VsZJ_AwB3X6r0WyVg1J3X69_QZ7LkbqVFx59At8hIvYFd_FOtNflnd7CcF_aPU |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA9jCupF9yF-m4MHBbO1Wdo0R5mOidsYOGG3kSavY5dWulbYf2_S1argxVvIIYSEx3sv-X0gdONAT5nOJyQAjkOYdjQRKpBE88g6TFptXFaYTfDJJJjPxbSG7isuDAAU4DPo2GHxl68Tldunsq5vil_XUsd3rHNWydaq2iuHuVvim0-JoHT-9SnpiO7ruG9RXKxTrvDLSqXIJIPD_-3hCLW_KXl4WiWbBqpB3EQHP9QEm6hRxuka35Zi0nctNKleDXBW1KiQ4tIpYomlUnkq1QaHG5yChcAvsak-TWBr_AhxsipmHvIssXKXFvLcRm-Dp1l_SEoPBbIyhUFGIgYmiYNQWlDXB2W1fFSoAvA0pz54XAoIbNKPqKt95mgZakkjHnHXDbkne8eoHicxnCAsNReB7lmNLI9FnIbMtNVAw4gFUprYPkUte1yL961MxqI8qbO_p6_R3nA2Hi1Gz5OXc7Rvb2eLkbtA9SzN4RLtqo9stU6vilv-BL6Cpx0 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Conference+proceedings+-+IEEE+International+Conference+on+Systems%2C+Man%2C+and+Cybernetics&rft.atitle=Improving+transfer+learning+accuracy+by+reusing+Stacked+Denoising+Autoencoders&rft.au=Kandaswamy%2C+Chetak&rft.au=Silva%2C+Luis+M.&rft.au=Alexandre%2C+Luis+A.&rft.au=Sousa%2C+Ricardo&rft.date=2014-10-01&rft.pub=IEEE&rft.issn=1062-922X&rft.spage=1380&rft.epage=1387&rft_id=info:doi/10.1109%2FSMC.2014.6974107&rft.externalDocID=6974107 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1062-922X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1062-922X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1062-922X&client=summon |