Sampling can be faster than optimization
Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years. There is, however, limited theoretical understanding of the relationships between these 2 kinds of methodology...
Uloženo v:
| Vydáno v: | Proceedings of the National Academy of Sciences - PNAS Ročník 116; číslo 42; s. 20881 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
United States
15.10.2019
|
| Témata: | |
| ISSN: | 1091-6490, 1091-6490 |
| On-line přístup: | Zjistit podrobnosti o přístupu |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years. There is, however, limited theoretical understanding of the relationships between these 2 kinds of methodology, and limited understanding of relative strengths and weaknesses. Moreover, existing results have been obtained primarily in the setting of convex functions (for optimization) and log-concave functions (for sampling). In this setting, where local properties determine global properties, optimization algorithms are unsurprisingly more efficient computationally than sampling algorithms. We instead examine a class of nonconvex objective functions that arise in mixture modeling and multistable systems. In this nonconvex setting, we find that the computational complexity of sampling algorithms scales linearly with the model dimension while that of optimization algorithms scales exponentially. |
|---|---|
| AbstractList | Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years. There is, however, limited theoretical understanding of the relationships between these 2 kinds of methodology, and limited understanding of relative strengths and weaknesses. Moreover, existing results have been obtained primarily in the setting of convex functions (for optimization) and log-concave functions (for sampling). In this setting, where local properties determine global properties, optimization algorithms are unsurprisingly more efficient computationally than sampling algorithms. We instead examine a class of nonconvex objective functions that arise in mixture modeling and multistable systems. In this nonconvex setting, we find that the computational complexity of sampling algorithms scales linearly with the model dimension while that of optimization algorithms scales exponentially.Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years. There is, however, limited theoretical understanding of the relationships between these 2 kinds of methodology, and limited understanding of relative strengths and weaknesses. Moreover, existing results have been obtained primarily in the setting of convex functions (for optimization) and log-concave functions (for sampling). In this setting, where local properties determine global properties, optimization algorithms are unsurprisingly more efficient computationally than sampling algorithms. We instead examine a class of nonconvex objective functions that arise in mixture modeling and multistable systems. In this nonconvex setting, we find that the computational complexity of sampling algorithms scales linearly with the model dimension while that of optimization algorithms scales exponentially. Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical machine learning in recent years. There is, however, limited theoretical understanding of the relationships between these 2 kinds of methodology, and limited understanding of relative strengths and weaknesses. Moreover, existing results have been obtained primarily in the setting of convex functions (for optimization) and log-concave functions (for sampling). In this setting, where local properties determine global properties, optimization algorithms are unsurprisingly more efficient computationally than sampling algorithms. We instead examine a class of nonconvex objective functions that arise in mixture modeling and multistable systems. In this nonconvex setting, we find that the computational complexity of sampling algorithms scales linearly with the model dimension while that of optimization algorithms scales exponentially. |
| Author | Chen, Yuansi Jordan, Michael I Ma, Yi-An Jin, Chi Flammarion, Nicolas |
| Author_xml | – sequence: 1 givenname: Yi-An orcidid: 0000-0001-6074-6638 surname: Ma fullname: Ma, Yi-An organization: Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720 – sequence: 2 givenname: Yuansi orcidid: 0000-0002-8899-7380 surname: Chen fullname: Chen, Yuansi organization: Department of Statistics, University of California, Berkeley, CA 94720 – sequence: 3 givenname: Chi surname: Jin fullname: Jin, Chi organization: Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720 – sequence: 4 givenname: Nicolas surname: Flammarion fullname: Flammarion, Nicolas organization: Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720 – sequence: 5 givenname: Michael I orcidid: 0000-0001-8935-817X surname: Jordan fullname: Jordan, Michael I email: jordan@cs.berkeley.edu organization: Department of Statistics, University of California, Berkeley, CA 94720 |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/31570618$$D View this record in MEDLINE/PubMed |
| BookMark | eNpNjztLxEAUhQdZcR9a20nKbbLOnUlyc0tZfMGChVqHeWokmcRMUuivN-AKVuc78HHgrNkidMExdgl8BxzldR9U3EEpOOcSoDhhK-AEaZERX_zjJVvH-DFLlJf8jC0l5MgLKFds-6zavqnDW2JUSLRLvIqjG5Lxfa5dP9Zt_a3Gugvn7NSrJrqLY27Y693ty_4hPTzdP-5vDqnJkMZUKAOQWa08odSWa10q5OAJygKkzclqRGfIZZnODXpHYGyZofLS-wJJbNj2d7cfus_JxbFq62hc06jguilWQhBhgSjErF4d1Um3zlb9ULdq-Kr-3okfs69S1g |
| CitedBy_id | crossref_primary_10_1016_j_physd_2021_132844 crossref_primary_10_3390_math10060898 crossref_primary_10_1007_s10107_023_01988_8 crossref_primary_10_1109_TIT_2024_3522494 crossref_primary_10_1016_j_jcp_2025_114208 crossref_primary_10_1109_LRA_2024_3455782 crossref_primary_10_1016_j_automatica_2023_111221 crossref_primary_10_1007_s11222_021_10054_2 crossref_primary_10_1109_TETCI_2021_3083428 crossref_primary_10_1214_25_EJS2397 crossref_primary_10_1214_22_BA1331 crossref_primary_10_1007_s11222_023_10220_8 crossref_primary_10_1038_s41598_025_12998_1 crossref_primary_10_3390_hydrology11020014 crossref_primary_10_1016_j_asr_2021_10_008 crossref_primary_10_1016_j_compstruct_2023_117657 crossref_primary_10_3390_e25081234 crossref_primary_10_1109_TIT_2023_3318152 crossref_primary_10_1088_1742_5468_ad01b2 crossref_primary_10_1017_jpr_2025_10025 crossref_primary_10_1093_imanum_draf045 crossref_primary_10_1109_TRPMS_2022_3148373 crossref_primary_10_1186_s12874_024_02235_0 crossref_primary_10_1137_24M1633765 crossref_primary_10_1088_1361_6420_ac9582 crossref_primary_10_1080_24725854_2023_2184884 crossref_primary_10_1109_TWC_2023_3342618 crossref_primary_10_1190_geo2021_0776_1 crossref_primary_10_1038_s42005_021_00768_0 crossref_primary_10_1109_TSP_2024_3516502 crossref_primary_10_1214_22_BJPS538 crossref_primary_10_1038_s41598_021_90144_3 crossref_primary_10_1007_s00180_023_01411_y crossref_primary_10_1007_s40304_023_00350_w crossref_primary_10_1109_TCOMM_2024_3435340 crossref_primary_10_1109_TASE_2024_3475951 crossref_primary_10_3150_21_BEJ1343 crossref_primary_10_3390_axioms11090462 crossref_primary_10_1109_JPROC_2024_3437730 crossref_primary_10_1109_TAES_2023_3320634 crossref_primary_10_1371_journal_pcbi_1010340 crossref_primary_10_1214_23_AAP2021 crossref_primary_10_1016_j_spa_2025_104684 crossref_primary_10_1177_17456916241258951 crossref_primary_10_1109_TRO_2025_3551198 crossref_primary_10_1016_j_tws_2022_110096 crossref_primary_10_1093_jrsssb_qkaf009 |
| ContentType | Journal Article |
| Copyright | Copyright © 2019 the Author(s). Published by PNAS. |
| Copyright_xml | – notice: Copyright © 2019 the Author(s). Published by PNAS. |
| DBID | NPM 7X8 |
| DOI | 10.1073/pnas.1820003116 |
| DatabaseName | PubMed MEDLINE - Academic |
| DatabaseTitle | PubMed MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic PubMed |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | no_fulltext_linktorsrc |
| Discipline | Sciences (General) |
| EISSN | 1091-6490 |
| ExternalDocumentID | 31570618 |
| Genre | Journal Article |
| GroupedDBID | --- -DZ -~X .55 0R~ 123 29P 2AX 2FS 2WC 4.4 53G 5RE 5VS 85S AACGO AAFWJ AANCE ABBHK ABOCM ABPLY ABPPZ ABTLG ABXSQ ABZEH ACGOD ACHIC ACIWK ACNCT ACPRK ADQXQ ADULT AENEX AEUPB AEXZC AFFNX AFOSN AFRAH ALMA_UNASSIGNED_HOLDINGS AQVQM BKOMP CS3 D0L DCCCD DIK DU5 E3Z EBS EJD F5P FRP GX1 H13 HH5 HYE IPSME JAAYA JBMMH JENOY JHFFW JKQEH JLS JLXEF JPM JSG JST KQ8 L7B LU7 N9A NPM N~3 O9- OK1 PNE PQQKQ R.V RHI RNA RNS RPM RXW SA0 SJN TAE TN5 UKR W8F WH7 WOQ WOW X7M XSW Y6R YBH YKV YSK ZCA ~02 ~KM 7X8 |
| ID | FETCH-LOGICAL-c479t-2ac114dbaf973bd0bb8a701f918613d59db77ec9e44b5c7fe91cd847af3ff6792 |
| IEDL.DBID | 7X8 |
| ISICitedReferencesCount | 96 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000490183000018&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1091-6490 |
| IngestDate | Thu Sep 04 16:31:01 EDT 2025 Thu Apr 03 06:56:30 EDT 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 42 |
| Keywords | Langevin Monte Carlo nonconvex optimization computational complexity |
| Language | English |
| License | Copyright © 2019 the Author(s). Published by PNAS. |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c479t-2ac114dbaf973bd0bb8a701f918613d59db77ec9e44b5c7fe91cd847af3ff6792 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| ORCID | 0000-0002-8899-7380 0000-0001-6074-6638 0000-0001-8935-817X |
| OpenAccessLink | https://www.pnas.org/doi/10.1073/pnas.1820003116 |
| PMID | 31570618 |
| PQID | 2299767722 |
| PQPubID | 23479 |
| ParticipantIDs | proquest_miscellaneous_2299767722 pubmed_primary_31570618 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-10-15 |
| PublicationDateYYYYMMDD | 2019-10-15 |
| PublicationDate_xml | – month: 10 year: 2019 text: 2019-10-15 day: 15 |
| PublicationDecade | 2010 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Proceedings of the National Academy of Sciences - PNAS |
| PublicationTitleAlternate | Proc Natl Acad Sci U S A |
| PublicationYear | 2019 |
| SSID | ssj0009580 |
| Score | 2.6698806 |
| Snippet | Optimization algorithms and Monte Carlo sampling algorithms have provided the computational foundations for the rapid growth in applications of statistical... |
| SourceID | proquest pubmed |
| SourceType | Aggregation Database Index Database |
| StartPage | 20881 |
| Title | Sampling can be faster than optimization |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/31570618 https://www.proquest.com/docview/2299767722 |
| Volume | 116 |
| WOSCitedRecordID | wos000490183000018&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LSwMxEA5qPXhR67O-iOChHmI32exmcxIRiwctBRV6K3mKB3erW_39TnZT9CIIXnILhMlkvi8zyTcInRXKaAPASpROFOGAuERnzBMOZNxpr7nwjWT-nRiNislEjmPCrY7PKhcxsQnUtjIhRz5gEDdFDlyQXc7eSOgaFaqrsYXGMuqkQGWCV4tJ8UN0t2jVCCQlOZfJQtpHpINZqeqLIF4evJrmv_PLBmeGG_9d4SZajwwTX7Uu0UVLrtxC3XiGa9yPQtPn26j_oMJ78vIZg32xdtirIJuAQzYdVxBLXuMnzR30NLx5vL4lsXMCMVzIOWHKwD3HauWlSLVNtC6USKiXtAD4tpm0WghnpONcZ0Z4J6mxgFPKp97nQrJdtFJWpdtHmNlcC-sTp1ioUDIN_EtnObfUZUmqbA-dLqwxBc8M5QZVuuqjnn7bo4f2WpNOZ62ExjSlmQAmURz8YfYhWgOWIgNg0OwIdTycS3eMVs3n_KV-P2m2HMbR-P4LYxC1qQ |
| linkProvider | ProQuest |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sampling+can+be+faster+than+optimization&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+-+PNAS&rft.au=Ma%2C+Yi-An&rft.au=Chen%2C+Yuansi&rft.au=Jin%2C+Chi&rft.au=Flammarion%2C+Nicolas&rft.date=2019-10-15&rft.issn=1091-6490&rft.eissn=1091-6490&rft.volume=116&rft.issue=42&rft.spage=20881&rft_id=info:doi/10.1073%2Fpnas.1820003116&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1091-6490&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1091-6490&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1091-6490&client=summon |