Distributed Identification of Top-l Inner Product Elements and its Application in a Peer-to-Peer Network

The inner product measures how closely two feature vectors are related. It is an important primitive for many popular data mining tasks, for example, clustering, classification, correlation computation, and decision tree construction. If the entire data set is available at a single site, then comput...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on knowledge and data engineering Vol. 20; no. 4; pp. 475 - 488
Main Authors: Das, K., Bhaduri, K., Kun Liu, Kargupta, H.
Format: Journal Article
Language:English
Published: New York, NY IEEE 01.04.2008
IEEE Computer Society
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Algorithms for data and knowledge management
Applied sciences
Classification tree analysis
Computation
Computer science; control theory; systems
Computer systems and distributed systems. User interface
Data mining
Data processing. List processing. Character string processing
Decision trees
Exact sciences and technology
Knowledge management applications
Large-scale systems
Mathematical analysis
Memory organisation. Data processing
Mining methods and algorithms
Network servers
Networks
Partitioning algorithms
Peer to peer computing
Probabilistic algorithms
Scalability
Software
Statistical distributions
Statistics
Studies
Vectors (mathematics)
Probabilistic algorithms
Knowledge management applications
Algorithms for data and knowledge management
Data mining
Mining methods and algorithms
Cluster analysis
Distributed data mining
Data analysis
Correlation
Peer to peer
Statistical analysis
Probabilistic approach
Scalability
Matrix product
pier-to-peer network
Information extraction
Distributed system
Decision tree
Modeling
Classification
inner product
Localization
ISSN:1041-4347, 1558-2191
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The inner product measures how closely two feature vectors are related. It is an important primitive for many popular data mining tasks, for example, clustering, classification, correlation computation, and decision tree construction. If the entire data set is available at a single site, then computing the inner product matrix and identifying the top (in terms of magnitude) entries is trivial. However, in many real-world scenarios, data is distributed across many locations and transmitting the data to a central server would be quite communication intensive and not scalable. This paper presents an approximate local algorithm for identifying top-l, inner products among pairs of feature vectors in a large asynchronous distributed environment such as a peer-to-peer (P2P) network. We develop a probabilistic algorithm for this purpose using order statistics and the Hoeffding bound. We present experimental results to show the effectiveness and scalability of the algorithm. Finally, we demonstrate an application of this technique for interest-based community formation in a P2P environment.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ObjectType-Article-2
ObjectType-Feature-1
content type line 23
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2007.190714