Optimal errors and phase transitions in high-dimensional generalized linear models
Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural n...
Uloženo v:
| Vydáno v: | Proceedings of the National Academy of Sciences - PNAS Ročník 116; číslo 12; s. 5451 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
United States
19.03.2019
|
| Témata: | |
| ISSN: | 1091-6490, 1091-6490 |
| On-line přístup: | Zjistit podrobnosti o přístupu |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or "free entropy") from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms. |
|---|---|
| AbstractList | Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or "free entropy") from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms.Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or "free entropy") from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms. Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-correcting codes, or benchmark models in neural networks. We evaluate the mutual information (or "free entropy") from which we deduce the Bayes-optimal estimation and generalization errors. Our analysis applies to the high-dimensional limit where both the number of samples and the dimension are large and their ratio is fixed. Nonrigorous predictions for the optimal errors existed for special cases of GLMs, e.g., for the perceptron, in the field of statistical physics based on the so-called replica method. Our present paper rigorously establishes those decades-old conjectures and brings forward their algorithmic interpretation in terms of performance of the generalized approximate message-passing algorithm. Furthermore, we tightly characterize, for many learning problems, regions of parameters for which this algorithm achieves the optimal performance and locate the associated sharp phase transitions separating learnable and nonlearnable regions. We believe that this random version of GLMs can serve as a challenging benchmark for multipurpose algorithms. |
| Author | Miolane, Léo Krzakala, Florent Macris, Nicolas Zdeborová, Lenka Barbier, Jean |
| Author_xml | – sequence: 1 givenname: Jean surname: Barbier fullname: Barbier, Jean email: jbarbier@ictp.it, leo.miolane@gmail.com organization: Laboratoire de Physique de l'Ecole Normale Supérieure, Université Paris-Sciences-et-Lettres, Centre National de la Recherche Scientifique, Sorbonne Université, Université Paris-Diderot, Sorbonne Paris Cité, 75005 Paris, France – sequence: 2 givenname: Florent surname: Krzakala fullname: Krzakala, Florent organization: Laboratoire de Physique de l'Ecole Normale Supérieure, Université Paris-Sciences-et-Lettres, Centre National de la Recherche Scientifique, Sorbonne Université, Université Paris-Diderot, Sorbonne Paris Cité, 75005 Paris, France – sequence: 3 givenname: Nicolas orcidid: 0000-0003-2189-7411 surname: Macris fullname: Macris, Nicolas organization: Communication Theory Laboratory, School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne, CH-1015 Lausanne, Switzerland – sequence: 4 givenname: Léo surname: Miolane fullname: Miolane, Léo email: jbarbier@ictp.it, leo.miolane@gmail.com organization: Département d'Informatique de l'Ecole Normale Supérieure, Université Paris-Sciences-et-Lettres, Centre National de la Recherche Scientifique, Inria, 75005 Paris, France; jbarbier@ictp.it leo.miolane@gmail.com – sequence: 5 givenname: Lenka surname: Zdeborová fullname: Zdeborová, Lenka organization: Institut de Physique Théorique, Centre National de la Recherche Scientifique et Commissariat à l'Energie Atomique, Université Paris-Saclay, 91191 Gif-sur-Yvette, France |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/30824595$$D View this record in MEDLINE/PubMed |
| BookMark | eNpNUEtLxDAYDLLiPvTsTXL00vVL0rTJURZfsLAgei7Z5utuJE1r0h7011twBU8zDDMDM0syC11AQq4ZrBmU4q4PJq2ZAl6CZKw4IwsGmmVFrmH2j8_JMqUPANBSwQWZC1A8l1ouyOuuH1xrPMUYu5ioCZb2R5OQDtGE5AbXhURdoEd3OGbWtTiJXZgCBwwYjXffaKl3AU2kbWfRp0ty3hif8OqEK_L--PC2ec62u6eXzf02q6VkQ4ZoG1NjaaG2VhrYcy2KOm8EFEogy4UtZdE0RpSWa62U1ZCLGnXJjFbF3vIVuf3t7WP3OWIaqtalGr03AbsxVZypUgouQUzWm5N13Ldoqz5Om-NX9fcD_wG8cGL3 |
| CitedBy_id | crossref_primary_10_1007_s10955_020_02495_2 crossref_primary_10_1088_1742_5468_ab43d2 crossref_primary_10_1007_s00440_021_01092_y crossref_primary_10_1214_20_AOS2038 crossref_primary_10_1088_1751_8121_acb530 crossref_primary_10_1007_s10208_021_09531_x crossref_primary_10_1109_TIT_2024_3481956 crossref_primary_10_1109_TCOMM_2021_3094509 crossref_primary_10_1088_1742_5468_ac9828 crossref_primary_10_1088_2632_2153_ace60f crossref_primary_10_1088_1751_8121_ab8416 crossref_primary_10_1109_TCOMM_2024_3395337 crossref_primary_10_1109_TIT_2024_3415654 crossref_primary_10_1109_TIT_2022_3225802 crossref_primary_10_1109_TIT_2020_2990880 crossref_primary_10_1073_pnas_2311810121 crossref_primary_10_1103_PhysRevResearch_7_013081 crossref_primary_10_1214_23_AOS2309 crossref_primary_10_1109_TIT_2023_3307553 crossref_primary_10_1214_24_AOS2449 crossref_primary_10_1214_23_AOS2301 crossref_primary_10_1214_21_AOS2100 crossref_primary_10_1007_s00220_022_04387_w crossref_primary_10_1088_1742_5468_ac59ac crossref_primary_10_1109_TIT_2025_3560070 crossref_primary_10_1103_PhysRevX_15_021085 crossref_primary_10_1214_25_AOS2489 crossref_primary_10_1038_s41467_023_36657_z crossref_primary_10_1109_TIT_2023_3321575 crossref_primary_10_1088_1751_8121_ab2735 crossref_primary_10_1109_TIT_2022_3182018 crossref_primary_10_1109_TIT_2024_3396472 crossref_primary_10_1103_PhysRevResearch_4_013201 crossref_primary_10_1109_TIT_2020_3015173 crossref_primary_10_1016_j_physa_2022_128154 crossref_primary_10_1016_j_physleta_2021_127756 crossref_primary_10_1088_1742_5468_ac3ae6 crossref_primary_10_1088_1742_5468_adfa75 crossref_primary_10_1088_1742_5468_ada696 crossref_primary_10_1088_1742_5468_adde3e crossref_primary_10_1109_TNNLS_2021_3050422 crossref_primary_10_1111_stan_12274 crossref_primary_10_1088_1751_8121_ab59ef crossref_primary_10_1088_1751_8121_ab8ff4 crossref_primary_10_1109_TIT_2022_3163342 crossref_primary_10_3150_21_BEJ1401 crossref_primary_10_1088_1742_5468_ababff crossref_primary_10_1088_2632_2153_acb428 crossref_primary_10_1093_imaiai_iaaf019 crossref_primary_10_1016_j_acha_2024_101736 crossref_primary_10_1088_1742_5468_ab3430 crossref_primary_10_1088_1742_5468_ade137 crossref_primary_10_1073_pnas_2302028120 crossref_primary_10_1103_PhysRevX_10_011057 crossref_primary_10_1103_PhysRevE_106_025304 crossref_primary_10_1109_TIT_2023_3299490 crossref_primary_10_1088_1742_5468_ab4bbb crossref_primary_10_1088_1742_5468_ac7e4c crossref_primary_10_1073_pnas_2403682121 crossref_primary_10_1088_1742_5468_ac2edd crossref_primary_10_1109_TWC_2021_3139034 crossref_primary_10_1093_imaiai_iaaf024 crossref_primary_10_1088_2632_2153_ac0615 crossref_primary_10_1109_TIT_2024_3455228 crossref_primary_10_1103_PhysRevX_10_041044 crossref_primary_10_1088_1742_5468_adb1d6 crossref_primary_10_1088_1742_5468_abc61d crossref_primary_10_1007_s10955_019_02470_6 crossref_primary_10_1093_jrsssb_qkae039 crossref_primary_10_1088_1751_8121_ad2c26 crossref_primary_10_1088_1742_5468_ad65e6 crossref_primary_10_1088_1751_8121_aba028 crossref_primary_10_1109_TIT_2023_3311408 crossref_primary_10_1088_2632_2153_acd749 crossref_primary_10_1093_imaiai_iaae024 crossref_primary_10_1088_1742_5468_ad65e1 crossref_primary_10_1109_TSP_2020_2974711 crossref_primary_10_1088_1742_5468_ac3a75 crossref_primary_10_1093_imaiai_iaab027 crossref_primary_10_1109_TIT_2024_3449321 crossref_primary_10_1103_l4p2_vrxt crossref_primary_10_1088_2632_2153_abfbbb crossref_primary_10_1007_s00220_022_04630_4 crossref_primary_10_1214_25_AAP2186 crossref_primary_10_1088_2632_2153_ac7d3b crossref_primary_10_1109_TIT_2024_3486685 crossref_primary_10_1088_1742_5468_ac9cc8 crossref_primary_10_1214_25_STS993 crossref_primary_10_1109_TIT_2023_3325187 crossref_primary_10_1088_1742_5468_add1ce crossref_primary_10_1088_1742_5468_ab7123 crossref_primary_10_1109_TIT_2024_3458953 crossref_primary_10_1109_TIT_2025_3535923 crossref_primary_10_1080_03610926_2024_2322616 crossref_primary_10_1109_TIT_2020_3033985 crossref_primary_10_1038_s42005_024_01870_9 crossref_primary_10_1214_21_AIHP1183 crossref_primary_10_1109_TCOMM_2023_3294960 crossref_primary_10_1214_24_AAP2120 crossref_primary_10_1214_22_AAP1874 |
| ContentType | Journal Article |
| Copyright | Copyright © 2019 the Author(s). Published by PNAS. |
| Copyright_xml | – notice: Copyright © 2019 the Author(s). Published by PNAS. |
| DBID | NPM 7X8 |
| DOI | 10.1073/pnas.1802705116 |
| DatabaseName | PubMed MEDLINE - Academic |
| DatabaseTitle | PubMed MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic PubMed |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | no_fulltext_linktorsrc |
| Discipline | Sciences (General) |
| EISSN | 1091-6490 |
| ExternalDocumentID | 30824595 |
| Genre | Research Support, Non-U.S. Gov't Journal Article |
| GroupedDBID | --- -DZ -~X .55 0R~ 123 29P 2AX 2FS 2WC 4.4 53G 5RE 5VS 85S AACGO AAFWJ AANCE ABBHK ABOCM ABPLY ABPPZ ABTLG ABXSQ ABZEH ACGOD ACHIC ACIWK ACNCT ACPRK ADQXQ ADULT AENEX AEUPB AEXZC AFFNX AFOSN AFRAH ALMA_UNASSIGNED_HOLDINGS AQVQM BKOMP CS3 D0L DCCCD DIK DU5 E3Z EBS EJD F5P FRP GX1 H13 HH5 HYE IPSME JAAYA JBMMH JENOY JHFFW JKQEH JLS JLXEF JPM JSG JST KQ8 L7B LU7 N9A NPM N~3 O9- OK1 PNE PQQKQ R.V RHI RNA RNS RPM RXW SA0 SJN TAE TN5 UKR W8F WH7 WOQ WOW X7M XSW Y6R YBH YKV YSK ZCA ~02 ~KM 7X8 |
| ID | FETCH-LOGICAL-c551t-eedface7d0cdd5a0b2936c4f30683e143d756ffa37d29988d9043ce971a986bd2 |
| IEDL.DBID | 7X8 |
| ISICitedReferencesCount | 177 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000461679000043&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1091-6490 |
| IngestDate | Sun Nov 09 10:32:19 EST 2025 Thu Apr 03 07:00:09 EDT 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 12 |
| Keywords | generalized linear model high-dimensional inference approximate message-passing algorithm perceptron Bayesian inference |
| Language | English |
| License | Copyright © 2019 the Author(s). Published by PNAS. |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c551t-eedface7d0cdd5a0b2936c4f30683e143d756ffa37d29988d9043ce971a986bd2 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| ORCID | 0000-0003-2189-7411 |
| OpenAccessLink | https://cea.hal.science/cea-01614258 |
| PMID | 30824595 |
| PQID | 2187532503 |
| PQPubID | 23479 |
| ParticipantIDs | proquest_miscellaneous_2187532503 pubmed_primary_30824595 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-03-19 |
| PublicationDateYYYYMMDD | 2019-03-19 |
| PublicationDate_xml | – month: 03 year: 2019 text: 2019-03-19 day: 19 |
| PublicationDecade | 2010 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Proceedings of the National Academy of Sciences - PNAS |
| PublicationTitleAlternate | Proc Natl Acad Sci U S A |
| PublicationYear | 2019 |
| SSID | ssj0009580 |
| Score | 2.6915462 |
| Snippet | Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze... |
| SourceID | proquest pubmed |
| SourceType | Aggregation Database Index Database |
| StartPage | 5451 |
| Title | Optimal errors and phase transitions in high-dimensional generalized linear models |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/30824595 https://www.proquest.com/docview/2187532503 |
| Volume | 116 |
| WOSCitedRecordID | wos000461679000043&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV09T8MwELWAMrAA5bN8yUgMMBgSkvhjQghRMUCpEEjdKse-QAeSkBQGfj3nOBUsSEgsmZwoOj_fe7ZP7wg5AmfIIlPOMuQeFqPoZRq4YVkaSgSUTTM_07diMJCjkRq2B251W1Y5y4lNoraFcWfkZ0hFqKyRsKOL8o25rlHudrVtoTFPOhFKGYdqMZI_THeldyNQIeOxCmbWPiI6K3Ndnzr3M4GoDPnv-rLhmf7Kf_9wlSy3CpNeekh0yRzka6TbruGaHrdG0yfr5OEe88UrjoWqKqqa6tzS8gVpjU4dg_liLjrJqfM0Ztb1AfAeHvTZf2PyCZY6naor2rTUqTfIU__68eqGtT0WmEGtNGVIkZk2IGxgrE10kCL9cxNnuJOQEaCYsiLhWaYjYZG4pLQqiCMDSoRaSZ7a802ykBc5bBMKJgVQsVAgUeYEICOtQVqBO8iIy1j3yOEsbmPEsLuY0DkU7_X4O3I9suWDPy692cbY2enEiUp2_vD2LllCPaNciVio9kgnwxUM-2TRfEwndXXQgAOfg-HdF7K7xS0 |
| linkProvider | ProQuest |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimal+errors+and+phase+transitions+in+high-dimensional+generalized+linear+models&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+-+PNAS&rft.au=Barbier%2C+Jean&rft.au=Krzakala%2C+Florent&rft.au=Macris%2C+Nicolas&rft.au=Miolane%2C+L%C3%A9o&rft.date=2019-03-19&rft.issn=1091-6490&rft.eissn=1091-6490&rft.volume=116&rft.issue=12&rft.spage=5451&rft_id=info:doi/10.1073%2Fpnas.1802705116&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1091-6490&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1091-6490&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1091-6490&client=summon |