Discrete Sparse Coding
Sparse coding algorithms with continuous latent variables have been the subject of a large number of studies. However, discrete latent spaces for sparse coding have been largely ignored. In this work, we study sparse coding with latents described by discrete instead of continuous prior distributions...
Uložené v:
| Vydané v: | Neural computation Ročník 29; číslo 11; s. 2979 |
|---|---|
| Hlavní autori: | , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
United States
01.11.2017
|
| ISSN: | 1530-888X, 1530-888X |
| On-line prístup: | Zistit podrobnosti o prístupe |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Sparse coding algorithms with continuous latent variables have been the subject of a large number of studies. However, discrete latent spaces for sparse coding have been largely ignored. In this work, we study sparse coding with latents described by discrete instead of continuous prior distributions. We consider the general case in which the latents (while being sparse) can take on any value of a finite set of possible values and in which we learn the prior probability of any value from data. This approach can be applied to any data generated by discrete causes, and it can be applied as an approximation of continuous causes. As the prior probabilities are learned, the approach then allows for estimating the prior shape without assuming specific functional forms. To efficiently train the parameters of our probabilistic generative model, we apply a truncated expectation-maximization approach (expectation truncation) that we modify to work with a general discrete prior. We evaluate the performance of the algorithm by applying it to a variety of tasks: (1) we use artificial data to verify that the algorithm can recover the generating parameters from a random initialization, (2) use image patches of natural images and discuss the role of the prior for the extraction of image components, (3) use extracellular recordings of neurons to present a novel method of analysis for spiking neurons that includes an intuitive discretization strategy, and (4) apply the algorithm on the task of encoding audio waveforms of human speech. The diverse set of numerical experiments presented in this letter suggests that discrete sparse coding algorithms can scale efficiently to work with realistic data sets and provide novel statistical quantities to describe the structure of the data. |
|---|---|
| AbstractList | Sparse coding algorithms with continuous latent variables have been the subject of a large number of studies. However, discrete latent spaces for sparse coding have been largely ignored. In this work, we study sparse coding with latents described by discrete instead of continuous prior distributions. We consider the general case in which the latents (while being sparse) can take on any value of a finite set of possible values and in which we learn the prior probability of any value from data. This approach can be applied to any data generated by discrete causes, and it can be applied as an approximation of continuous causes. As the prior probabilities are learned, the approach then allows for estimating the prior shape without assuming specific functional forms. To efficiently train the parameters of our probabilistic generative model, we apply a truncated expectation-maximization approach (expectation truncation) that we modify to work with a general discrete prior. We evaluate the performance of the algorithm by applying it to a variety of tasks: (1) we use artificial data to verify that the algorithm can recover the generating parameters from a random initialization, (2) use image patches of natural images and discuss the role of the prior for the extraction of image components, (3) use extracellular recordings of neurons to present a novel method of analysis for spiking neurons that includes an intuitive discretization strategy, and (4) apply the algorithm on the task of encoding audio waveforms of human speech. The diverse set of numerical experiments presented in this letter suggests that discrete sparse coding algorithms can scale efficiently to work with realistic data sets and provide novel statistical quantities to describe the structure of the data.Sparse coding algorithms with continuous latent variables have been the subject of a large number of studies. However, discrete latent spaces for sparse coding have been largely ignored. In this work, we study sparse coding with latents described by discrete instead of continuous prior distributions. We consider the general case in which the latents (while being sparse) can take on any value of a finite set of possible values and in which we learn the prior probability of any value from data. This approach can be applied to any data generated by discrete causes, and it can be applied as an approximation of continuous causes. As the prior probabilities are learned, the approach then allows for estimating the prior shape without assuming specific functional forms. To efficiently train the parameters of our probabilistic generative model, we apply a truncated expectation-maximization approach (expectation truncation) that we modify to work with a general discrete prior. We evaluate the performance of the algorithm by applying it to a variety of tasks: (1) we use artificial data to verify that the algorithm can recover the generating parameters from a random initialization, (2) use image patches of natural images and discuss the role of the prior for the extraction of image components, (3) use extracellular recordings of neurons to present a novel method of analysis for spiking neurons that includes an intuitive discretization strategy, and (4) apply the algorithm on the task of encoding audio waveforms of human speech. The diverse set of numerical experiments presented in this letter suggests that discrete sparse coding algorithms can scale efficiently to work with realistic data sets and provide novel statistical quantities to describe the structure of the data. Sparse coding algorithms with continuous latent variables have been the subject of a large number of studies. However, discrete latent spaces for sparse coding have been largely ignored. In this work, we study sparse coding with latents described by discrete instead of continuous prior distributions. We consider the general case in which the latents (while being sparse) can take on any value of a finite set of possible values and in which we learn the prior probability of any value from data. This approach can be applied to any data generated by discrete causes, and it can be applied as an approximation of continuous causes. As the prior probabilities are learned, the approach then allows for estimating the prior shape without assuming specific functional forms. To efficiently train the parameters of our probabilistic generative model, we apply a truncated expectation-maximization approach (expectation truncation) that we modify to work with a general discrete prior. We evaluate the performance of the algorithm by applying it to a variety of tasks: (1) we use artificial data to verify that the algorithm can recover the generating parameters from a random initialization, (2) use image patches of natural images and discuss the role of the prior for the extraction of image components, (3) use extracellular recordings of neurons to present a novel method of analysis for spiking neurons that includes an intuitive discretization strategy, and (4) apply the algorithm on the task of encoding audio waveforms of human speech. The diverse set of numerical experiments presented in this letter suggests that discrete sparse coding algorithms can scale efficiently to work with realistic data sets and provide novel statistical quantities to describe the structure of the data. |
| Author | Lücke, Jörg Exarchakis, Georgios |
| Author_xml | – sequence: 1 givenname: Georgios surname: Exarchakis fullname: Exarchakis, Georgios email: georgios.exarchakis@uol.de organization: Machine Learning Lab, Cluster of Excellence Hearing4all and Department for Medical Physics and Acoustics, Carl-von-Ossietzky University Oldenburg, 26111 Oldenburg, Germany georgios.exarchakis@uol.de – sequence: 2 givenname: Jörg surname: Lücke fullname: Lücke, Jörg email: joerg.luecke@uol.de organization: Machine Learning Lab, Cluster of Excellence Hearing4all and Department for Medical Physics and Acoustics, Carl-von-Ossietzky University Oldenburg, 26111 Oldenburg, Germany joerg.luecke@uol.de |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/28957027$$D View this record in MEDLINE/PubMed |
| BookMark | eNpNjztLxEAUhQdZcR_aiaVsaRO9d543pcQnLFioYDeMkxuJbB5mksJ_74IrWJ1TfHycsxSztmtZiFOES0Qrr1qOnQ8eENAciAUaBRkRvc3-9blYpvQJABbBHIm5pNw4kG4hzm7qFAceef3chyHxuujKuv04FodV2CY-2edKvN7dvhQP2ebp_rG43mRRaxozDsExlcpZqShnayrDgUIOUZN1rAjYBWPBqWgrnSOBthGcIQJpoAxyJS5-vf3QfU2cRt_s9vB2G1rupuQx10ZKhYg79HyPTu8Nl74f6iYM3_7vi_wBUYRKSA |
| CitedBy_id | crossref_primary_10_3390_math11122674 crossref_primary_10_1088_2634_4386_ac970d |
| ContentType | Journal Article |
| DBID | NPM 7X8 |
| DOI | 10.1162/neco_a_01015 |
| DatabaseName | PubMed MEDLINE - Academic |
| DatabaseTitle | PubMed MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic PubMed |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | no_fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1530-888X |
| ExternalDocumentID | 28957027 |
| Genre | Research Support, Non-U.S. Gov't Journal Article |
| GroupedDBID | --- -~X .4S .DC 0R~ 123 36B 4.4 41~ 53G 6IK AAJGR AALMD AAYOK ABAZT ABDBF ABDNZ ABEFU ABIVO ABJNI ACGFO ACUHS ACYGS ADIYS ADMLS AEGXH AEILP AENEX AIAGR ALMA_UNASSIGNED_HOLDINGS ARCSS AVWKF AZFZN BEFXN BFFAM BGNUA BKEBE BPEOZ CAG COF CS3 DU5 EAP EAS EBC EBD EBS ECS EDO EJD EMB EMK EMOBN EPL EPS EST ESX F5P FEDTE FNEHJ HVGLF HZ~ H~9 I-F IPLJI JAVBF MCG MINIK MKJ NPM O9- OCL P2P PK0 PQQKQ RMI SV3 TUS WG8 WH7 XJE ZWS 7X8 ABUFD ABVLG AMVHM |
| ID | FETCH-LOGICAL-c448t-eaa7e8d3762389e65f5ea8a90c4867e380e7a56073c6f4918046c075880250da2 |
| IEDL.DBID | 7X8 |
| ISICitedReferencesCount | 8 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000413292600005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1530-888X |
| IngestDate | Sun Nov 09 13:59:25 EST 2025 Thu Apr 03 07:05:38 EDT 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 11 |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c448t-eaa7e8d3762389e65f5ea8a90c4867e380e7a56073c6f4918046c075880250da2 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| OpenAccessLink | https://direct.mit.edu/neco/article-pdf/29/11/2979/1026079/neco_a_01015.pdf |
| PMID | 28957027 |
| PQID | 1945223111 |
| PQPubID | 23479 |
| ParticipantIDs | proquest_miscellaneous_1945223111 pubmed_primary_28957027 |
| PublicationCentury | 2000 |
| PublicationDate | 2017-11-00 20171101 |
| PublicationDateYYYYMMDD | 2017-11-01 |
| PublicationDate_xml | – month: 11 year: 2017 text: 2017-11-00 |
| PublicationDecade | 2010 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Neural computation |
| PublicationTitleAlternate | Neural Comput |
| PublicationYear | 2017 |
| SSID | ssj0006105 |
| Score | 2.299804 |
| Snippet | Sparse coding algorithms with continuous latent variables have been the subject of a large number of studies. However, discrete latent spaces for sparse coding... |
| SourceID | proquest pubmed |
| SourceType | Aggregation Database Index Database |
| StartPage | 2979 |
| Title | Discrete Sparse Coding |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/28957027 https://www.proquest.com/docview/1945223111 |
| Volume | 29 |
| WOSCitedRecordID | wos000413292600005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LS8NAEB7UevBifVXriwhelzbZ7CMnkWrxoKXgg9zCupmFXpJoq7_f2TzoSRC85BbIJjvzfd_O5BuA6yjCRHOXMynimMXGGEYgGDG0zuXev0zVFd23RzWb6TRN5u2B27Jtq-xyYp2o89L6M_IRiW2iCpxC86b6YH5qlK-utiM0NqHHicr4li6Vrt3CZdPCSEE9ZqT00q7xXUajgsRdZjLvsCZ-J5c1yEz7_328Pdht6WVw2-yHfdjA4gD63eiGoI3kQxjcLShfEGEOniuSthhMSg9iR_A6vX-ZPLB2RAKzpKtWDI1RqHPKEgS9CUrhBBptkrH1TnrI9RiVIVKjuJUuTkJNctgSS6CoJe6Tm2gAW0VZ4AkEEkn6hNG7jYWLQxKXjgsrtbJcUj7PwyFcdSvPaAv6uoIpsPxaZuu1D-G4eX1Z1XhlZKTnhCLpe_qHu89gJ_KgWf_pdw49RwGIF7Btv1eL5edl_W3pOps__QBT4Kuq |
| linkProvider | ProQuest |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Discrete+Sparse+Coding&rft.jtitle=Neural+computation&rft.au=Exarchakis%2C+Georgios&rft.au=L%C3%BCcke%2C+J%C3%B6rg&rft.date=2017-11-01&rft.issn=1530-888X&rft.eissn=1530-888X&rft.volume=29&rft.issue=11&rft.spage=2979&rft_id=info:doi/10.1162%2Fneco_a_01015&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1530-888X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1530-888X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1530-888X&client=summon |