Scalable detection of statistically significant communities and hierarchies, using message passing for modularity
Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory ''communities'' in random graphs where none...
Uložené v:
| Vydané v: | Proceedings of the National Academy of Sciences - PNAS Ročník 111; číslo 51; s. 18144 |
|---|---|
| Hlavní autori: | , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
United States
23.12.2014
|
| Predmet: | |
| ISSN: | 1091-6490, 1091-6490 |
| On-line prístup: | Zistit podrobnosti o prístupe |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory ''communities'' in random graphs where none exist. We address this problem by using the modularity as a Hamiltonian at finite temperature and using an efficient belief propagation algorithm to obtain the consensus of many partitions with high modularity, rather than looking for a single partition that maximizes it. We show analytically and numerically that the proposed algorithm works all of the way down to the detectability transition in networks generated by the stochastic block model. It also performs well on real-world networks, revealing large communities in some networks where previous work has claimed no communities exist. Finally we show that by applying our algorithm recursively, subdividing communities until no statistically significant subcommunities can be found, we can detect hierarchical structure in real-world networks more efficiently than previous methods. |
|---|---|
| AbstractList | Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory ''communities'' in random graphs where none exist. We address this problem by using the modularity as a Hamiltonian at finite temperature and using an efficient belief propagation algorithm to obtain the consensus of many partitions with high modularity, rather than looking for a single partition that maximizes it. We show analytically and numerically that the proposed algorithm works all of the way down to the detectability transition in networks generated by the stochastic block model. It also performs well on real-world networks, revealing large communities in some networks where previous work has claimed no communities exist. Finally we show that by applying our algorithm recursively, subdividing communities until no statistically significant subcommunities can be found, we can detect hierarchical structure in real-world networks more efficiently than previous methods.Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory ''communities'' in random graphs where none exist. We address this problem by using the modularity as a Hamiltonian at finite temperature and using an efficient belief propagation algorithm to obtain the consensus of many partitions with high modularity, rather than looking for a single partition that maximizes it. We show analytically and numerically that the proposed algorithm works all of the way down to the detectability transition in networks generated by the stochastic block model. It also performs well on real-world networks, revealing large communities in some networks where previous work has claimed no communities exist. Finally we show that by applying our algorithm recursively, subdividing communities until no statistically significant subcommunities can be found, we can detect hierarchical structure in real-world networks more efficiently than previous methods. Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory ''communities'' in random graphs where none exist. We address this problem by using the modularity as a Hamiltonian at finite temperature and using an efficient belief propagation algorithm to obtain the consensus of many partitions with high modularity, rather than looking for a single partition that maximizes it. We show analytically and numerically that the proposed algorithm works all of the way down to the detectability transition in networks generated by the stochastic block model. It also performs well on real-world networks, revealing large communities in some networks where previous work has claimed no communities exist. Finally we show that by applying our algorithm recursively, subdividing communities until no statistically significant subcommunities can be found, we can detect hierarchical structure in real-world networks more efficiently than previous methods. |
| Author | Zhang, Pan Moore, Cristopher |
| Author_xml | – sequence: 1 givenname: Pan surname: Zhang fullname: Zhang, Pan organization: Santa Fe Institute, Santa Fe, NM 87501 – sequence: 2 givenname: Cristopher surname: Moore fullname: Moore, Cristopher email: moore@santafe.edu organization: Santa Fe Institute, Santa Fe, NM 87501 moore@santafe.edu |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/25489096$$D View this record in MEDLINE/PubMed |
| BookMark | eNpNkD1PwzAYhC0Eoh8wsyGPDKTYju3YI6r4kioxAHP01rGLUWKnsTP03xNBkZjuOd3phlug0xCDReiKkhUlVXnXB0gryomuKkIpPUFzSjQtJNfk9B_P0CKlL0KIFoqcoxkTXGmi5Rzt3wy0sG0tbmy2JvsYcHQ4Zcg-ZT-F7QEnvwveTSZkbGLXjcFnbxOG0OBPbwcYzCTpFo_Jhx3ubEqws7iH9ONdHHAXm7GFwefDBTpz0CZ7edQl-nh8eF8_F5vXp5f1_aYwkvFcGABWNoIK3ghWOcGZEGCVLEtoVGXAgBZSM-cqM-GWSstLJThxyklWUsaW6OZ3tx_ifrQp151PxrYtBBvHVFPJCVdEKD1Vr4_VcdvZpu4H38FwqP9uYt96I25u |
| CitedBy_id | crossref_primary_10_1088_1367_2630_ab73ca crossref_primary_10_1109_TKDE_2019_2911585 crossref_primary_10_1038_s41467_021_22559_5 crossref_primary_10_1162_NETN_a_00002 crossref_primary_10_1080_23307706_2022_2146010 crossref_primary_10_1007_s11467_017_0657_y crossref_primary_10_1038_s41598_019_44892_y crossref_primary_10_1145_3596605 crossref_primary_10_1016_j_ins_2025_122421 crossref_primary_10_1038_s42005_022_00816_3 crossref_primary_10_1109_TNNLS_2022_3201906 crossref_primary_10_1088_1742_5468_2016_05_053301 crossref_primary_10_1145_3512962 crossref_primary_10_1371_journal_pone_0215296 crossref_primary_10_1103_PhysRevResearch_2_043271 crossref_primary_10_1088_1742_6596_699_1_012015 crossref_primary_10_1016_j_physa_2022_127722 crossref_primary_10_1007_s12293_022_00370_z crossref_primary_10_1007_s41109_022_00466_y crossref_primary_10_1088_1742_5468_aaa8f5 crossref_primary_10_1186_s40537_022_00627_x crossref_primary_10_1371_journal_pone_0191604 crossref_primary_10_1109_TKDE_2021_3104155 crossref_primary_10_1111_gean_12278 crossref_primary_10_1063_5_0268930 crossref_primary_10_1016_j_eswa_2022_117794 crossref_primary_10_1109_TNSE_2020_3008538 crossref_primary_10_1109_TCBB_2019_2912602 crossref_primary_10_1016_j_neucom_2024_128169 crossref_primary_10_1137_17M1138972 crossref_primary_10_1103_PhysRevX_11_021003 crossref_primary_10_1109_TKDE_2021_3125330 crossref_primary_10_1109_TKDE_2016_2563425 crossref_primary_10_1137_19M1270446 crossref_primary_10_1007_s13369_022_06747_z crossref_primary_10_1016_j_jep_2016_07_014 crossref_primary_10_1016_j_patcog_2024_110487 crossref_primary_10_1038_s41598_018_21352_7 crossref_primary_10_1038_s41598_018_29725_8 crossref_primary_10_1016_j_ipm_2024_104053 crossref_primary_10_1016_j_physa_2024_130274 crossref_primary_10_1007_s40484_018_0153_6 crossref_primary_10_1155_2018_8098325 crossref_primary_10_1088_1367_2630_aab547 crossref_primary_10_1109_TKDE_2019_2914200 crossref_primary_10_1186_s13059_023_03062_0 crossref_primary_10_1007_s10618_020_00681_0 crossref_primary_10_1371_journal_pcsy_0000062 crossref_primary_10_1016_j_chaos_2025_117249 crossref_primary_10_1109_TBDATA_2022_3223644 crossref_primary_10_1080_00018732_2016_1211393 crossref_primary_10_1038_s41598_021_99175_2 crossref_primary_10_1038_s41467_024_52355_w crossref_primary_10_1038_s41598_019_46380_9 crossref_primary_10_1007_s11579_020_00273_y crossref_primary_10_1038_s41598_017_03623_x crossref_primary_10_1016_j_cmpb_2025_108681 crossref_primary_10_1073_pnas_2321112121 crossref_primary_10_1038_s41540_018_0052_5 crossref_primary_10_1371_journal_pone_0254057 crossref_primary_10_1038_s41598_024_55190_7 crossref_primary_10_1146_annurev_psych_122414_033634 crossref_primary_10_1073_pnas_1611275114 crossref_primary_10_1109_TKDE_2022_3197612 crossref_primary_10_1038_s41598_024_54594_9 crossref_primary_10_4018_IJSWIS_341232 crossref_primary_10_1088_1742_5468_2016_09_093403 crossref_primary_10_1038_s41467_022_34341_2 crossref_primary_10_1088_2399_6528_aa9b6b crossref_primary_10_1038_s41598_017_05585_6 crossref_primary_10_1016_j_physrep_2016_09_002 crossref_primary_10_1016_j_asoc_2019_106010 crossref_primary_10_1103_PhysRevResearch_2_033325 crossref_primary_10_1109_TKDE_2019_2911943 crossref_primary_10_1007_s10115_024_02097_4 crossref_primary_10_1038_s41467_021_24884_1 crossref_primary_10_1088_1674_1056_26_3_038902 crossref_primary_10_1111_rssb_12505 crossref_primary_10_1109_TVCG_2024_3456167 crossref_primary_10_1016_j_jocs_2024_102283 crossref_primary_10_1109_TBDATA_2021_3104005 crossref_primary_10_1093_comnet_cnx004 crossref_primary_10_1016_j_neucom_2024_127703 crossref_primary_10_1109_TKDE_2019_2958806 crossref_primary_10_1109_TNNLS_2024_3367873 crossref_primary_10_1016_j_scs_2021_103508 crossref_primary_10_1007_s41060_024_00631_9 crossref_primary_10_1109_TKDE_2020_3015667 crossref_primary_10_1016_j_chaos_2016_02_004 crossref_primary_10_1103_PhysRevResearch_7_013065 crossref_primary_10_1038_s41598_019_54708_8 crossref_primary_10_1109_TBDATA_2020_2964544 crossref_primary_10_1038_s41598_018_25560_z crossref_primary_10_1016_j_neucom_2020_04_067 crossref_primary_10_1080_19420889_2017_1296614 crossref_primary_10_3233_IFS_151974 crossref_primary_10_1088_1742_5468_2015_11_P11006 |
| ContentType | Journal Article |
| DBID | CGR CUY CVF ECM EIF NPM 7X8 |
| DOI | 10.1073/pnas.1409770111 |
| DatabaseName | Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed MEDLINE - Academic |
| DatabaseTitle | MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic MEDLINE |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | no_fulltext_linktorsrc |
| Discipline | Sciences (General) |
| EISSN | 1091-6490 |
| ExternalDocumentID | 25489096 |
| Genre | Research Support, Non-U.S. Gov't Journal Article |
| GroupedDBID | --- -DZ -~X .55 0R~ 123 29P 2AX 2FS 2WC 4.4 53G 5RE 5VS 85S AACGO AAFWJ AANCE ABBHK ABOCM ABPLY ABPPZ ABTLG ABXSQ ABZEH ACGOD ACHIC ACIWK ACNCT ACPRK ADQXQ ADULT AENEX AEUPB AEXZC AFFNX AFOSN AFRAH ALMA_UNASSIGNED_HOLDINGS AQVQM BKOMP CGR CS3 CUY CVF D0L DCCCD DIK DU5 E3Z EBS ECM EIF EJD F5P FRP GX1 H13 HH5 HYE IPSME JAAYA JBMMH JENOY JHFFW JKQEH JLS JLXEF JPM JSG JST KQ8 L7B LU7 N9A NPM N~3 O9- OK1 PNE PQQKQ R.V RHI RNA RNS RPM RXW SA0 SJN TAE TN5 UKR W8F WH7 WOQ WOW X7M XSW Y6R YBH YKV YSK ZCA ~02 ~KM 7X8 ADXHL |
| ID | FETCH-LOGICAL-c624t-caa23d5154d527f54255ae8633ad87caca95692ff7cca9b16e438540f8f623122 |
| IEDL.DBID | 7X8 |
| ISICitedReferencesCount | 131 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000346767200032&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1091-6490 |
| IngestDate | Thu Sep 04 18:01:06 EDT 2025 Thu Apr 03 07:00:40 EDT 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 51 |
| Keywords | phase transitions statistical significance networks community detection message-passing algorithms |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c624t-caa23d5154d527f54255ae8633ad87caca95692ff7cca9b16e438540f8f623122 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| OpenAccessLink | https://www.pnas.org/content/pnas/111/51/18144.full.pdf |
| PMID | 25489096 |
| PQID | 1640480589 |
| PQPubID | 23479 |
| ParticipantIDs | proquest_miscellaneous_1640480589 pubmed_primary_25489096 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-12-23 |
| PublicationDateYYYYMMDD | 2014-12-23 |
| PublicationDate_xml | – month: 12 year: 2014 text: 2014-12-23 day: 23 |
| PublicationDecade | 2010 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Proceedings of the National Academy of Sciences - PNAS |
| PublicationTitleAlternate | Proc Natl Acad Sci U S A |
| PublicationYear | 2014 |
| References | 20481789 - Phys Rev E Stat Nonlin Soft Matter Phys. 2010 Apr;81(4 Pt 2):046110 24277835 - Proc Natl Acad Sci U S A. 2013 Dec 24;110(52):20935-40 18999496 - Phys Rev E Stat Nonlin Soft Matter Phys. 2008 Oct;78(4 Pt 2):046110 26167197 - J Stat Mech. 2014 May;2014(5):null 22304154 - Phys Rev E Stat Nonlin Soft Matter Phys. 2011 Dec;84(6 Pt 2):066106 21405744 - Phys Rev E Stat Nonlin Soft Matter Phys. 2011 Jan;83(1 Pt 2):016107 17025705 - Phys Rev E Stat Nonlin Soft Matter Phys. 2006 Sep;74(3 Pt 2):036104 18216267 - Proc Natl Acad Sci U S A. 2008 Jan 29;105(4):1118-23 16196754 - Phys Rev E Stat Nonlin Soft Matter Phys. 2005 Aug;72(2 Pt 2):027104 15244693 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Jun;69(6 Pt 2):066133 15447530 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Aug;70(2 Pt 2):025101 20481785 - Phys Rev E Stat Nonlin Soft Matter Phys. 2010 Apr;81(4 Pt 2):046106 17025687 - Phys Rev E Stat Nonlin Soft Matter Phys. 2006 Sep;74(3 Pt 2):035102 22468223 - Sci Rep. 2012;2:336 14995526 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Feb;69(2 Pt 2):026113 15697438 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Dec;70(6 Pt 2):066111 18451861 - Nature. 2008 May 1;453(7191):98-101 16907154 - Phys Rev E Stat Nonlin Soft Matter Phys. 2006 Jul;74(1 Pt 2):016110 21559480 - PLoS One. 2011;6(4):e18961 21902340 - Phys Rev Lett. 2011 Aug 5;107(6):065701 17190818 - Proc Natl Acad Sci U S A. 2007 Jan 2;104(1):36-41 17881571 - Proc Natl Acad Sci U S A. 2007 Sep 25;104(39):15224-9 |
| References_xml | – reference: 18216267 - Proc Natl Acad Sci U S A. 2008 Jan 29;105(4):1118-23 – reference: 21559480 - PLoS One. 2011;6(4):e18961 – reference: 15244693 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Jun;69(6 Pt 2):066133 – reference: 24277835 - Proc Natl Acad Sci U S A. 2013 Dec 24;110(52):20935-40 – reference: 17190818 - Proc Natl Acad Sci U S A. 2007 Jan 2;104(1):36-41 – reference: 16196754 - Phys Rev E Stat Nonlin Soft Matter Phys. 2005 Aug;72(2 Pt 2):027104 – reference: 17025705 - Phys Rev E Stat Nonlin Soft Matter Phys. 2006 Sep;74(3 Pt 2):036104 – reference: 20481789 - Phys Rev E Stat Nonlin Soft Matter Phys. 2010 Apr;81(4 Pt 2):046110 – reference: 15697438 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Dec;70(6 Pt 2):066111 – reference: 22304154 - Phys Rev E Stat Nonlin Soft Matter Phys. 2011 Dec;84(6 Pt 2):066106 – reference: 26167197 - J Stat Mech. 2014 May;2014(5):null – reference: 15447530 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Aug;70(2 Pt 2):025101 – reference: 22468223 - Sci Rep. 2012;2:336 – reference: 17881571 - Proc Natl Acad Sci U S A. 2007 Sep 25;104(39):15224-9 – reference: 21902340 - Phys Rev Lett. 2011 Aug 5;107(6):065701 – reference: 17025687 - Phys Rev E Stat Nonlin Soft Matter Phys. 2006 Sep;74(3 Pt 2):035102 – reference: 18999496 - Phys Rev E Stat Nonlin Soft Matter Phys. 2008 Oct;78(4 Pt 2):046110 – reference: 16907154 - Phys Rev E Stat Nonlin Soft Matter Phys. 2006 Jul;74(1 Pt 2):016110 – reference: 21405744 - Phys Rev E Stat Nonlin Soft Matter Phys. 2011 Jan;83(1 Pt 2):016107 – reference: 20481785 - Phys Rev E Stat Nonlin Soft Matter Phys. 2010 Apr;81(4 Pt 2):046106 – reference: 14995526 - Phys Rev E Stat Nonlin Soft Matter Phys. 2004 Feb;69(2 Pt 2):026113 – reference: 18451861 - Nature. 2008 May 1;453(7191):98-101 |
| SSID | ssj0009580 |
| Score | 2.566612 |
| Snippet | Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same... |
| SourceID | proquest pubmed |
| SourceType | Aggregation Database Index Database |
| StartPage | 18144 |
| SubjectTerms | Algorithms Cluster Analysis Models, Theoretical Stochastic Processes |
| Title | Scalable detection of statistically significant communities and hierarchies, using message passing for modularity |
| URI | https://www.ncbi.nlm.nih.gov/pubmed/25489096 https://www.proquest.com/docview/1640480589 |
| Volume | 111 |
| WOSCitedRecordID | wos000346767200032&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1LS8NAEF7UevCi1md9sYIHBdcmu2myexIRixdLQYXeynYfvbRJa6rgv3dmk6IXQZBAIIdA2Hnkm92Z7yPkwkfh8sxJa1jihWOAQiLGVcRHACGMiU0Qm8h6PTkYqH694VbWbZXLnBgStS0M7pG3Adbj-HNHqtvZnKFqFJ6u1hIaq6QhAMqgV2cD-YN0V1ZsBCpmaaKiJbVPJtqzXJc3gewpQ7X13_Fl-M90t_77hdtks0aY9K5yiSZZcfkOadYxXNLLmmj6apfMn8FAODpFrVuElqycFp7ijFGgb9aTySfFBg9sJwILUFNNkyAHK9W5paijHU4iXHlNsYN-TKcoqTJ2dAagHJ8BE9NpYbHZFfD-HnntPrzcP7JagoGZlCcLZrTmwgLmSWyHZ74DEd7RTqZCaCszo42G-kpx7zPwBDWKU5cICSDQSw-4KuZ8n6zlRe4OCY0h70YG8Mco9YnLIHGMoDo1UlnhFFiiRc6XyzoEF8dzC5274r0cfi9sixxUthnOKi6OIdS3UkEZdvSHt4_JBsCdQNPIxQlpeAhwd0rWzQes6ttZ8B249_pPX5aZ0H0 |
| linkProvider | ProQuest |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Scalable+detection+of+statistically+significant+communities+and+hierarchies%2C+using+message+passing+for+modularity&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+-+PNAS&rft.au=Zhang%2C+Pan&rft.au=Moore%2C+Cristopher&rft.date=2014-12-23&rft.eissn=1091-6490&rft.volume=111&rft.issue=51&rft.spage=18144&rft_id=info:doi/10.1073%2Fpnas.1409770111&rft_id=info%3Apmid%2F25489096&rft_id=info%3Apmid%2F25489096&rft.externalDocID=25489096 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1091-6490&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1091-6490&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1091-6490&client=summon |