LSH SimilarityJoin Pattern in FastFlow

Similarity joins are recognized to be among the most used data processing and analysis operations. We introduce a C++-based high-level parallel pattern implemented on top of FastFlow Building Blocks to provide the programmer with ready-to-use similarity join computations. The SimilarityJoin pattern...

Full description

Saved in:
Bibliographic Details
Published in:International journal of parallel programming Vol. 52; no. 3; pp. 207 - 230
Main Authors: Tonci, Nicolò, Rivault, Sébastien, Bamha, Mostafa, Robert, Sophie, Limet, Sébastien, Torquati, Massimo
Format: Journal Article
Language:English
Published: New York Springer US 01.06.2024
Springer Nature B.V
Springer Verlag
Subjects:
Algorithms
C++ (programming language)
Computer Science
Data processing
Data structures
Datasets
Distributed memory
Performance assessment
Processor Architectures
Similarity
Software Engineering/Programming and Operating Systems
Theory of Computation
LSH Similarity join
Distributed programming
Big data MapReduce
Parallel patterns
High-level parallel programming
Big data
MapReduce
ISSN:0885-7458, 1573-7640
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Similarity joins are recognized to be among the most used data processing and analysis operations. We introduce a C++-based high-level parallel pattern implemented on top of FastFlow Building Blocks to provide the programmer with ready-to-use similarity join computations. The SimilarityJoin pattern is implemented according to the MapReduce paradigm enriched with locality sensitive hashing (LSH) to optimize the whole computation. The new parallel pattern can be used with any C++ serializable data structure and executed on shared- and distributed-memory machines. We present experimental validations of the proposed solution considering two different clusters and small and large input datasets to evaluate in-core and out-of-core executions. The performance assessment of the SimilarityJoin pattern has been conducted by comparing the execution time against the one obtained from the original hand-tuned Hadoop-based implementation of the LSH-based similarity join algorithms as well as a Spark-based version. The experiments show that the SimilarityJoin pattern: (1) offers a significant performance improvement for small and medium datasets; (2) is competitive also for computations using large input datasets producing out-of-core executions.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0885-7458
1573-7640
DOI:10.1007/s10766-024-00772-1