Efficient and Scalable Algorithms for Inferring Likely Invariants in Distributed Systems
Distributed systems generate a large amount of monitoring data such as log files to track their operational status. However, it is hard to correlate such monitoring data effectively across distributed systems and along observation time for system management. In previous work, we proposed a concept n...
Uloženo v:
| Vydáno v: | IEEE transactions on knowledge and data engineering Ročník 19; číslo 11; s. 1508 - 1523 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
New York, NY
IEEE
01.11.2007
IEEE Computer Society The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Témata: | |
| ISSN: | 1041-4347, 1558-2191 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Distributed systems generate a large amount of monitoring data such as log files to track their operational status. However, it is hard to correlate such monitoring data effectively across distributed systems and along observation time for system management. In previous work, we proposed a concept named flow intensity to measure the intensity with which internal monitoring data reacts to the volume of user requests. We calculated flow intensity measurements from monitoring data and proposed an algorithm to automatically search constant relationships between flow intensities measured at various points across distributed systems. If such relationships hold all the time, we regard them as invariants of the underlying systems. Invariants can be used to characterize complex systems and support various system management tasks. However, the computational complexity of the previous invariant search algorithm is high so that it may not scale well in large systems with thousands of measurements. In this paper, we propose two efficient but approximate algorithms for inferring invariants in large-scale systems. The computational complexity of new randomized algorithms is significantly reduced, and experimental results from a real system are also included to demonstrate the accuracy and efficiency of our new algorithms. |
|---|---|
| AbstractList | Distributed systems generate a large amount of monitoring data such as log files to track their operational status. However, it is hard to correlate such monitoring data effectively across distributed systems and along observation time for system management. In previous work, we proposed a concept named flow intensity to measure the intensity with which internal monitoring data reacts to the volume of user requests. We calculated flow intensity measurements from monitoring data and proposed an algorithm to automatically search constant relationships between flow intensities measured at various points across distributed systems. If such relationships hold all the time, we regard them as invariants of the underlying systems. Invariants can be used to characterize complex systems and support various system management tasks. However, the computational complexity of the previous invariant search algorithm is high so that it may not scale well in large systems with thousands of measurements. In this paper, we propose two efficient but approximate algorithms for inferring invariants in large-scale systems. The computational complexity of new randomized algorithms is significantly reduced, and experimental results from a real system are also included to demonstrate the accuracy and efficiency of our new algorithms. The computational complexity of new randomized algorithms is significantly reduced, and experimental results from a real system are also included to demonstrate the accuracy and efficiency of our new algorithms. |
| Author | Yoshihira, K. Haifeng Chen Guofei Jiang |
| Author_xml | – sequence: 1 surname: Guofei Jiang fullname: Guofei Jiang organization: NEC Lab. America Inc., Princeton – sequence: 2 surname: Haifeng Chen fullname: Haifeng Chen organization: NEC Lab. America Inc., Princeton – sequence: 3 givenname: K. surname: Yoshihira fullname: Yoshihira, K. organization: NEC Lab. America Inc., Princeton |
| BackLink | http://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=19139488$$DView record in Pascal Francis |
| BookMark | eNp9kc1LXDEUxUOxULXuC908CtXVm-br5SVL0fGDDnRRhe7CfZkbG_smzyYZYf57M4y4EHR1L5ffOXDPOSB7cYpIyBdGZ4xR8-Pm5_l8xintZ8xQJfUHss-6TrecGbZXdypZK4XsP5GDnO8ppbrXbJ_8mXsfXMBYGojL5reDEYYRm9Pxbkqh_F3lxk-puY4eUwrxrlmEfzhu6uERUoBYchNicx5ySWFYF6wWm1xwlT-Tjx7GjEfP85DcXsxvzq7axa_L67PTReuEFqV1XCjHvFh6LkznhGFLbjxfMqA9Skm9UnJgvUHRAR1A9WoYOj2AAOSD6kEckpOd70Oa_q8xF7sK2eE4QsRpna3WVJlOGFPJ43dJITstuZEV_PYKvJ_WKdYvrGE1z473okLfnyHINTSfILqQ7UMKK0gbW1MXRmpdObXjXJpyTuitCwVKmGJJEEbLqN32Z7f92W1_dtdfFdJXwhfvtyVfd5KAiC-4FMJwpsQT9bCm6w |
| CODEN | ITKEEH |
| CitedBy_id | crossref_primary_10_1109_TSC_2020_2993251 crossref_primary_10_1145_3046946 crossref_primary_10_1016_j_jss_2013_03_060 crossref_primary_10_1007_s11432_013_4906_6 crossref_primary_10_1109_TSMC_2015_2430834 crossref_primary_10_1007_s10586_010_0120_0 crossref_primary_10_1109_TETC_2015_2497143 crossref_primary_10_1007_s10586_008_0063_x crossref_primary_10_1109_TSC_2017_2679715 crossref_primary_10_1109_TDSC_2021_3083671 crossref_primary_10_1109_TIM_2021_3139696 crossref_primary_10_1109_TSE_2016_2623623 crossref_primary_10_1145_2094091_2094101 crossref_primary_10_1145_2601436 |
| Cites_doi | 10.1145/1029894.1029901 10.1145/233008.233034 10.1201/9780203756126 10.1017/CBO9780511814075 10.1109/ADL.1998.670376 10.1109/32.908957 10.1016/S0967-0661(97)00053-1 10.1109/DSN.2006.70 10.1109/2.901170 10.1145/502512.502584 10.1145/1095809.1095821 10.1145/581376.581377 10.1109/TDSC.2006.52 10.1109/TKDE.2005.103 10.1109/ICAC.2006.1662399 10.1145/846183.846188 10.1007/3-540-44934-5_9 10.1145/352871.352878 |
| ContentType | Journal Article |
| Copyright | 2007 INIST-CNRS Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2007 |
| Copyright_xml | – notice: 2007 INIST-CNRS – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2007 |
| DBID | 97E RIA RIE AAYXX CITATION IQODW 7SC 7SP 8FD JQ2 L7M L~C L~D F28 FR3 |
| DOI | 10.1109/TKDE.2007.190648 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Pascal-Francis Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional ANTE: Abstracts in New Technology & Engineering Engineering Research Database |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional Engineering Research Database ANTE: Abstracts in New Technology & Engineering |
| DatabaseTitleList | Technology Research Database Technology Research Database Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science Applied Sciences |
| EISSN | 1558-2191 |
| EndPage | 1523 |
| ExternalDocumentID | 2544498811 19139488 10_1109_TKDE_2007_190648 4339216 |
| Genre | orig-research |
| GroupedDBID | -~X .DC 0R~ 1OL 29I 4.4 5GY 5VS 6IK 97E 9M8 AAJGR AARMG AASAJ AAWTH ABAZT ABFSI ABQJQ ABVLG ACGFO ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 E.L EBS EJD F5P HZ~ H~9 ICLAB IEDLZ IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNI RNS RXW RZB TAE TAF TN5 UHB VH1 AAYXX CITATION AAYOK IQODW RIG 7SC 7SP 8FD JQ2 L7M L~C L~D F28 FR3 |
| ID | FETCH-LOGICAL-c383t-c236c1f3df2395c391d29f2d1a07e440f664b179e35a0ba676bb58ba3ae2b67a3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 34 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000249563900006&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1041-4347 |
| IngestDate | Sat Sep 27 18:58:12 EDT 2025 Thu Oct 02 06:17:45 EDT 2025 Sun Nov 30 04:22:22 EST 2025 Wed Apr 02 07:15:28 EDT 2025 Sat Nov 29 08:09:48 EST 2025 Tue Nov 18 22:35:36 EST 2025 Wed Aug 27 02:52:16 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 11 |
| Keywords | Invariant data management Decision support system Complex system Time series Distributed system Randomized algorithm Computational complexity Search algorithm monitoring data Surveillance Storage management Time management Distributed systems randomized algorithms Algorithm complexity Log file Monitoring Large scale system invariants |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html CC BY 4.0 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c383t-c236c1f3df2395c391d29f2d1a07e440f664b179e35a0ba676bb58ba3ae2b67a3 |
| Notes | ObjectType-Article-2 SourceType-Scholarly Journals-1 ObjectType-Feature-1 content type line 14 content type line 23 |
| PQID | 912195273 |
| PQPubID | 23500 |
| PageCount | 16 |
| ParticipantIDs | proquest_miscellaneous_880695399 pascalfrancis_primary_19139488 proquest_journals_912195273 crossref_citationtrail_10_1109_TKDE_2007_190648 proquest_miscellaneous_34584294 ieee_primary_4339216 crossref_primary_10_1109_TKDE_2007_190648 |
| PublicationCentury | 2000 |
| PublicationDate | 2007-11-01 |
| PublicationDateYYYYMMDD | 2007-11-01 |
| PublicationDate_xml | – month: 11 year: 2007 text: 2007-11-01 day: 01 |
| PublicationDecade | 2000 |
| PublicationPlace | New York, NY |
| PublicationPlace_xml | – name: New York, NY – name: New York |
| PublicationTitle | IEEE transactions on knowledge and data engineering |
| PublicationTitleAbbrev | TKDE |
| PublicationYear | 2007 |
| Publisher | IEEE IEEE Computer Society The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: IEEE Computer Society – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 O’Madadhain (ref19) 2003 ref12 ref15 ref14 ref30 ref10 ref2 Patterson (ref21) 2002 ref1 ref17 ref18 Cormen (ref6) 1990 Brogan (ref4) 1990 ref26 Smith (ref24) 2003 ref25 Wolsey (ref27) 1999 ref22 Baeza-Yates (ref3) 1999 Oppenheimer (ref20) ref28 Hoxmeier (ref11) ref29 ref8 ref9 ref5 DeGroot (ref7) 2001 Ljung (ref16) 1998 |
| References_xml | – ident: ref22 doi: 10.1145/1029894.1029901 – ident: ref2 doi: 10.1145/233008.233034 – start-page: 140 volume-title: Proc. Sixth Americas Conf. Information Systems (AMCIS ’00) ident: ref11 article-title: System Response Time and User Satisfaction: An Experimental Study of Browser-Based Applications – volume-title: Math. of the Discrete Fourier Transform (DFT) year: 2003 ident: ref24 – ident: ref9 doi: 10.1201/9780203756126 – ident: ref18 doi: 10.1017/CBO9780511814075 – ident: ref29 doi: 10.1109/ADL.1998.670376 – ident: ref8 doi: 10.1109/32.908957 – volume-title: Modern Control Theory year: 1990 ident: ref4 – ident: ref12 doi: 10.1016/S0967-0661(97)00053-1 – start-page: 1 volume-title: Proc. Fourth Usenix Symp. Internet Technologies and Systems (USITS ’03) ident: ref20 article-title: Why Do Internet Services Fail, and What Can Be Done about It – ident: ref30 doi: 10.1109/DSN.2006.70 – ident: ref1 doi: 10.1109/2.901170 – volume-title: Introduction to Algorithms year: 1990 ident: ref6 – volume-title: Technical Report UCI-ICS 03-17, UC Irvine, Dept. of Information and Computer Science year: 2003 ident: ref19 article-title: The Jung (Java Universal Network/Graph) Framework – ident: ref28 doi: 10.1145/502512.502584 – volume-title: Modern Information Retrieval year: 1999 ident: ref3 – ident: ref5 doi: 10.1145/1095809.1095821 – ident: ref10 doi: 10.1145/581376.581377 – ident: ref14 doi: 10.1109/TDSC.2006.52 – volume-title: System Identification—Theory for The User year: 1998 ident: ref16 – ident: ref15 doi: 10.1109/TKDE.2005.103 – volume-title: Technical Report UCB//CSD-02-1175 year: 2002 ident: ref21 article-title: Recovery-Oriented Computing (ROC): Motivation, Definition, Techniques, and Case Studies – volume-title: Probability and Statistics year: 2001 ident: ref7 – ident: ref13 doi: 10.1109/ICAC.2006.1662399 – ident: ref26 doi: 10.1145/846183.846188 – ident: ref25 doi: 10.1007/3-540-44934-5_9 – ident: ref17 doi: 10.1145/352871.352878 – volume-title: Integer and Combinatorial Optimization year: 1999 ident: ref27 |
| SSID | ssj0008781 |
| Score | 2.084546 |
| Snippet | Distributed systems generate a large amount of monitoring data such as log files to track their operational status. However, it is hard to correlate such... The computational complexity of new randomized algorithms is significantly reduced, and experimental results from a real system are also included to... |
| SourceID | proquest pascalfrancis crossref ieee |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 1508 |
| SubjectTerms | Algorithms Algorithms for data and knowledge management Analysis of Algorithms and Problem Complexity Applied sciences Approximation Complexity Computation Computational complexity Computer networks Computer science; control theory; systems Computerized monitoring Costs Data mining Data processing. List processing. Character string processing Distributed Systems Exact sciences and technology Fault detection Fluid flow measurement Hardware Information systems. Data bases Invariants Large-scale systems Management information systems Memory organisation. Data processing Monitoring Software Studies System Management Systems management Time series analysis Volume measurement |
| Title | Efficient and Scalable Algorithms for Inferring Likely Invariants in Distributed Systems |
| URI | https://ieeexplore.ieee.org/document/4339216 https://www.proquest.com/docview/912195273 https://www.proquest.com/docview/34584294 https://www.proquest.com/docview/880695399 |
| Volume | 19 |
| WOSCitedRecordID | wos000249563900006&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-2191 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0008781 issn: 1041-4347 databaseCode: RIE dateStart: 19890101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3BbtQwEB21FQc4UNqCCC2tD1yQCHXixI6PVbsVCFRxKNLeorFjw4oliza7lfh7PE42FEErcYuSsRV5PGOPx_MewKus4YWruE21QJ4WMvcp0kZOWiOxKhuFzkeyCXV1VU2n-tMWvBlrYZxz8fKZe0uPMZffLOyajsqIC07nmdyGbaVkX6s1et1KRULSEF2EmEgUapOS5Pr0-sPFpAcrDKufJKafW0tQ5FShG5HYhUHxPZvFX445rjaXu__3n0_g8bCrZGf9NNiDLdfuw-6GsYENBrwPj27BDx7AdBLxI0JfDNsmSOGcCqnY2fzLYjlbff3esbClZe-pJpBasI-zb27-M7y4CRE2XaBhs5ZdEPQusWa50EWPf_4UPl9Ors_fpQPTQmpDhLpKbS6kzbxofC50aYXOmlz7vMmQK1cU3EtZmGC6TpTIDUoljSkrgwJdbqRC8Qx22kXrngMrlfLWeN24UheGVyi9dcIogWHjVnJM4HQz-LUdYMiJDWNex3CE65rUReyYqu7VlcDrscWPHoLjHtkDUscoN2gigeM_9Pu7H0JFDT4sgcONwuvBiLtaZ8GdE0BdAifj12B9lFLB1i3WXS0ozZzrIgF2h0Twj1IT_O-Lf__aITyMx8WxvPEIdlbLtXsJD-zNatYtj-Mc_wX58Pq5 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Rb9MwED6NMQl42NgGIgw2P-wFiTAnTuz4cdo6bVqpeChS3yLbsUdFSVHTTuLf43PSMMQ2ibcoOVuRz3f2-XzfB3CcVDSzBTWxZIrGGU9drHAjx43mqsgroawLZBNiNComE_llAz72tTDW2nD5zH7Cx5DLr-ZmhUdlyAUn04Q_gafInNVVa_V-txCBktTHFz4qYplYJyWpPBlfnw9auEK__nHk-rmzCAVWFbwTqRo_LK7ls_jHNYf15mLn__70JWx3-0py2k6EXdiw9R7srDkbSGfCe_DiDgDhPkwGAUHC90VUXXkpNcNSKnI6u5kvpstvPxriN7XkCqsCsQUZTr_b2S__4tbH2HiFhkxrco7gu8ibZX0XLQL6K_h6MRifXcYd10JsfIy6jE3KuEkcq1zKZG6YTKpUurRKFBU2y6jjPNPeeC3LFdWKC651XmjFlE01F4q9hs16Xts3QHIhnNFOVjaXmaaF4s5YpgVTfuuWUxXByXrwS9MBkSMfxqwMAQmVJaoL-TFF2aorgg99i58tCMcjsvuojl6u00QEh3_p908_iIvqvVgEB2uFl50ZN6VMvENHiLoIjvqv3v4wqaJqO181JcNEcyqzCMgDEt5DcokAwG_v_7UjeHY5_jwsh1ej6wN4Hg6PQ7HjO9hcLlb2PWyZ2-W0WRyG-f4bKNP-Ag |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Efficient+and+Scalable+Algorithms+for+Inferring+Likely+Invariants+in+Distributed+Systems&rft.jtitle=IEEE+transactions+on+knowledge+and+data+engineering&rft.au=Jiang%2C+Guofei&rft.au=Chen%2C+Haifeng&rft.au=Yoshihira%2C+K&rft.date=2007-11-01&rft.issn=1041-4347&rft.volume=19&rft.issue=11&rft.spage=1508&rft.epage=1523&rft_id=info:doi/10.1109%2FTKDE.2007.190648&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1041-4347&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1041-4347&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1041-4347&client=summon |