Zobrazit v EDS

Internet based shared memory in a distributed computing system

Uloženo v:
Podrobná bibliografie
Název: Internet based shared memory in a distributed computing system
Patent Number: 10437,673
Datum vydání: October 08, 2019
Appl. No: 16/183644
Application Filed: November 07, 2018
Abstrakt: A method includes sending, by a computing device of a distributed storage and task (DST) network, a plurality of sets of encoded data slices and index generation task information to a set of DST execution units. The method further includes receiving partial data indexes from the set of DST execution units. The method further includes generating a data index based on the partial data indexes and determining an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index. The method further includes partitioning the operational task into a set of partial tasks and sending the set of partial tasks to the set of DST execution units.
Inventors: International Business Machines Corporation (Armonk, NY, US)
Assignees: PURE STORAGE, INC. (Mountain View, CA, US)
Claim: 1. A method comprises: sending, by a computing device of a distributed storage and task (DST) network, a plurality of sets of encoded data slices and index generation task information to a set of DST execution units of the DST network, wherein raw data is dispersed storage error encoded in accordance with initial dispersed storage error encoding parameters to produce the plurality of sets of encoded data slices, and wherein the index generation task information is regarding the raw data; receiving, by the computing device, partial data indexes from the set of DST execution units, wherein the partial data indexes are generated from the plurality of sets of encoded data slices and based on the index generation task information; generating, by the computing device, a data index based on the partial data indexes; determining, by the computing device, an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the plurality of sets of encoded data slices and execute a data processing function on the data index; partitioning, by the computing device, the operational task into a set of partial tasks; and sending, by the computing device, the set of partial tasks to the set of DST execution units.
Claim: 2. The method of claim 1 further comprises: determining, by the computing device, the index generation task information based on a storage profile regarding the raw data.
Claim: 3. The method of claim 2 , wherein the index generation task information includes one or more of: a search parameter; a keyword; pattern recognition information; and timing information.
Claim: 4. The method of claim 2 , wherein the storage profile includes one or more of: a security requirement; a performance requirement; an estimated retrieval frequency; an estimated distributed processing level; a data visibility profile; a data owner; a data index storage indicator; and a data deletion policy.
Claim: 5. The method of claim 1 , wherein the data index includes one or more of: identified keywords; identified dates; identified internet protocol addresses; partial content; a word count; raw data statistics; a summary of the raw data; and one or more DST network addresses.
Claim: 6. The method of claim 1 further comprises: when the operational task is long term storage of the raw data, selecting, by the computing device, new dispersed storage error encoding parameters; dispersed storage error encoding, by the computing device, the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and sending, by the computing device, the new plurality of sets of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.
Claim: 7. A computing device of a distributed storage and task (DST) network, the computing device comprises: an interface; memory; and a processing module operably coupled to the memory and the interface, wherein the processing module is operable to: send a plurality of sets of encoded data slices and index generation task information to a set of DST execution units of the DST network, wherein raw data is dispersed storage error encoded in accordance with initial dispersed storage error encoding parameters to produce the plurality of sets of encoded data slices, and wherein the index generation task information is regarding the raw data; receiving, by the computing device, partial data indexes from the set of DST execution units, wherein the partial data indexes are generated from the plurality of sets of encoded data slices and based on the index generation task information; generating, by the computing device, a data index based on the partial data indexes; determining, by the computing device, an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the plurality of sets of encoded data slices and execute a data processing function on the data index; partitioning, by the computing device, the operational task into a set of partial tasks; and sending, by the computing device, the set of partial tasks to the set of DST execution units.
Claim: 8. The computing device of claim 7 , wherein the processing module is further operable to: determine the index generation task information based on a storage profile regarding the raw data.
Claim: 9. The computing device of claim 8 , wherein the index generation task information includes one or more of: a search parameter; a keyword; pattern recognition information; and timing information.
Claim: 10. The computing device of claim 8 , wherein the storage profile includes one or more of: a security requirement; a performance requirement; an estimated retrieval frequency; an estimated distributed processing level; a data visibility profile; a data owner; a data index storage indicator; and a data deletion policy.
Claim: 11. The computing device of claim 7 , wherein the data index includes one or more of: identified keywords; identified dates; identified internet protocol addresses; partial content; a word count; raw data statistics; a summary of the raw data; and one or more DST network addresses.
Claim: 12. The computing device of claim 7 , wherein the processing module is further operable to: when the operational task is long term storage of the raw data, select new dispersed storage error encoding parameters; dispersed storage error encode the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and send the new plurality of sets of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.
Patent References Cited: 1903750 April 1933 Bacher et al.
4092732 May 1978 Ouchi
5454101 September 1995 Mackay et al.
5485474 January 1996 Rabin
5774643 June 1998 Lubbers et al.
5802364 September 1998 Senator et al.
5809285 September 1998 Hilland
5890156 March 1999 Rekieta et al.
5987622 November 1999 Lo Verso et al.
5991414 November 1999 Garay et al.
6012159 January 2000 Fischer et al.
6058454 May 2000 Gerlach et al.
6128277 October 2000 Bruck et al.
6175571 January 2001 Haddock et al.
6192472 February 2001 Garay et al.
6256688 July 2001 Suetaka et al.
6272658 August 2001 Steele et al.
6301604 October 2001 Nojima
6356949 March 2002 Katsandres et al.
6366995 April 2002 Vilkov et al.
6374336 April 2002 Peters et al.
6415373 July 2002 Peters et al.
6418539 July 2002 Walker
6449688 September 2002 Peters et al.
6567948 May 2003 Steele et al.
6571282 May 2003 Bowman-Amuah
6609223 August 2003 Wolfgang
6718361 April 2004 Basani et al.
6760808 July 2004 Peters et al.
6785768 August 2004 Peters et al.
6785783 August 2004 Buckland
6826711 November 2004 Moulton et al.
6879596 April 2005 Dooply
7003688 February 2006 Pittelkow et al.
7024451 April 2006 Jorgenson
7024609 April 2006 Wolfgang et al.
7080101 July 2006 Watson et al.
7103824 September 2006 Halford
7103915 September 2006 Redlich et al.
7111115 September 2006 Peters et al.
7140044 November 2006 Redlich et al.
7146644 December 2006 Redlich et al.
7171493 January 2007 Shu et al.
7222133 May 2007 Raipurkar et al.
7240236 July 2007 Cutts et al.
7272613 September 2007 Sim et al.
7480909 January 2009 McKean et al.
7636724 December 2009 de la Torre et al.
8468368 June 2013 Gladwin et al.
8819011 August 2014 Motwani et al.
8832234 September 2014 Brooker et al.
9338140 May 2016 OHare et al.
2001/0033296 October 2001 Fullerton et al.
2002/0044657 April 2002 Asano et al.
2002/0062422 May 2002 Butterworth et al.
2002/0166079 November 2002 Ulrich et al.
2003/0018927 January 2003 Gadir et al.
2003/0037261 February 2003 Meffert et al.
2003/0065617 April 2003 Watkins et al.
2003/0084020 May 2003 Shu
2003/0163507 August 2003 Chang et al.
2004/0024963 February 2004 Talagala et al.
2004/0122917 June 2004 Menon et al.
2004/0215998 October 2004 Buxton et al.
2004/0228493 November 2004 Ma
2005/0018853 January 2005 Lain et al.
2005/0100022 May 2005 Ramprashad
2005/0114594 May 2005 Corbett et al.
2005/0125593 June 2005 Karpoff et al.
2005/0131993 June 2005 Fatula
2005/0132070 June 2005 Redlich et al.
2005/0144382 June 2005 Schmisseur
2005/0229069 October 2005 Hassner et al.
2006/0047907 March 2006 Shiga et al.
2006/0136448 June 2006 Cialini et al.
2006/0156059 July 2006 Kitamura
2006/0224603 October 2006 Correll
2007/0016604 January 2007 Murthy et al.
2007/0079081 April 2007 Gladwin et al.
2007/0079082 April 2007 Gladwin et al.
2007/0079083 April 2007 Gladwin et al.
2007/0088970 April 2007 Buxton et al.
2007/0174192 July 2007 Gladwin et al.
2007/0214285 September 2007 Au et al.
2007/0234110 October 2007 Soran et al.
2007/0283167 December 2007 Venters et al.
2008/0256183 October 2008 Flynn et al.
2009/0094251 April 2009 Gladwin et al.
2009/0094318 April 2009 Gladwin et al.
2009/0150605 June 2009 Flynn et al.
2010/0023524 January 2010 Gladwin et al.
2010/0287200 November 2010 Dhuse
2010/0332751 December 2010 Quigley et al.
2011/0071988 March 2011 Resch et al.
2011/0185258 July 2011 Grube et al.
2011/0264717 October 2011 Grube et al.
2011/0307645 December 2011 Hall
2012/0331088 December 2012 OHare et al.
2013/0238900 September 2013 Leggette et al.
2014/0040662 February 2014 Dhuse et al.
Other References: Chung; An Automatic Data Segmentation Method for 3D Measured Data Points; National Taiwan University; pp. 1-8; 1998. cited by applicant
Communication pursuant to Article 94(3) EPC; EP Application No. 12856963.9; dated May 26, 2017; 9 pages. cited by applicant
Harrison; Lightweight Directory Access Protocol (LDAP): Authentication Methods and Security Mechanisms; IETF Network Working Group; RFC 4513; Jun. 2006; pp. 1-32. cited by applicant
International Business Machines; Response to EP communication dated May 26, 2017; Nov. 1, 2017; 5 pgs. cited by applicant
International Search Report and Written Opinion; International Search Authority; International Application No. PCT/US12/68883; dated Feb. 26, 2013; 12 pgs. cited by applicant
Kubiatowicz, et al.; OceanStore: An Architecture for Global-Scale Persistent Storage; Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2000); Nov. 2000; pp. 1-12. cited by applicant
Legg; Lightweight Directory Access Protocol (LDAP): Syntaxes and Matching Rules; IETF Network Working Group; RFC 4517; Jun. 2006; pp. 1-50. cited by applicant
Plank, T1: Erasure Codes for Storage Applications; FAST2005, 4th Usenix Conference on File Storage Technologies; Dec. 13-16, 2005; pp. 1-74. cited by applicant
Rabin; Efficient Dispersal of Information for Security, Load Balancing, and Fault Tolerance; Journal of the Association for Computer Machinery; vol. 36, No. 2; Apr. 1989; pp. 335-348. cited by applicant
Satran, et al.; Internet Small Computer Systems Interface (iSCSI); IETF Network Working Group; RFC 3720; Apr. 2004; pp. 1-257. cited by applicant
Sciberras; Lightweight Directory Access Protocol (LDAP): Schema for User Applications; IETF Network Working Group; RFC 4519; Jun. 2006; pp. 1-33. cited by applicant
Sermersheim; Lightweight Directory Access Protocol (LDAP): The Protocol; IETF Network Working Group; RFC 4511; Jun. 2006; pp. 1-68. cited by applicant
Shamir; How to Share a Secret; Communications of the ACM; vol. 22, No. 11; Nov. 1979; pp. 612-613. cited by applicant
Smith; Lightweight Directory Access Protocol (LDAP): String Representation of Search Filters; IETF Network Working Group; RFC 4515; Jun. 2006; pp. 1-12. cited by applicant
Smith; Lightweight Directory Access Protocol (LDAP): Uniform Resource Locator; IETF Network Working Group; RFC 4516; Jun. 2006; pp. 1-15. cited by applicant
Wildi; Java iSCSi Initiator; Master Thesis; Department of Computer and Information Science, University of Konstanz; Feb. 2007; 60 pgs. cited by applicant
Xin, et al.; Evaluation of Distributed Recovery in Large-Scale Storage Systems; 13th IEEE International Symposium on High Performance Distributed Computing; Jun. 2004; pp. 172-181. cited by applicant
Zeilenga; Lightweight Directory Access Protocol (LDAP): Technical Specification Road Map; IETF Network Working Group; RFC 4510; Jun. 2006; pp. 1-8. cited by applicant
Zeilenga; Lightweight Directory Access Protocol (LDAP): Directory Information Models; IETF Network Working Group; RFC 4512; Jun. 2006; pp. 1-49. cited by applicant
Zeilenga; Lightweight Directory Access Protocol (LDAP): Internationalized String Preparation; IETF Network Working Group; RFC 4518; Jun. 2006; pp. 1-14. cited by applicant
Zeilenga; Lightweight Directory Access Protocol (LDAP): String Representation of Distinguished Names; IETF Network Working Group; RFC 4514; Jun. 2006; pp. 1-15. cited by applicant
Primary Examiner: Kanaan, Simon P
Attorney, Agent or Firm: Garlick & Markison
Markison, Timothy W.
Markison, Patricia A.
Přístupové číslo: edspgr.10437673
Databáze: USPTO Patent Grants
Popis
Abstrakt:A method includes sending, by a computing device of a distributed storage and task (DST) network, a plurality of sets of encoded data slices and index generation task information to a set of DST execution units. The method further includes receiving partial data indexes from the set of DST execution units. The method further includes generating a data index based on the partial data indexes and determining an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index. The method further includes partitioning the operational task into a set of partial tasks and sending the set of partial tasks to the set of DST execution units.