Articolo in rivista, 2022, ENG, 10.7717/PEERJ-CS.1058

FDup: a framework for general-purpose and efficient entity deduplication of record collections

De Bonis M.; Manghi P.; Atzori C.

CNR-ISTI, Pisa, Italy; CNR-ISTI, Pisa, Italy; CNR-ISTI, Pisa, Italy

Deduplication is a technique aiming at identifying and resolving duplicate metadata records in a collection. This article describes FDup (Flat Collections Deduper), a general-purpose software framework supporting a complete deduplication workflow to manage big data record collections: metadata record data model definition, identification of candidate duplicates, identification of duplicates. FDup brings two main innovations: first, it delivers a full deduplication framework in a single easy-to-use software package based on Apache Spark Hadoop framework, where developers can customize the optimal and parallel workflow steps of blocking, sliding windows, and similarity matching function via an intuitive configuration file; second, it introduces a novel approach to improve performance, beyond the known techniques of "blocking" and "sliding window", by introducing a smart similarity matching function T-match. T-match is engineered as a decision tree that drives the comparisons of the fields of two records as branches of predicates and allows for successful or unsuccessful early-exit strategies. The efficacy of the approach is proved by experiments performed over big data collections of metadata records in the OpenAIRE Research Graph, a known open access knowledge base in Scholarly communication.

PeerJ Computer Science 8 (e1058)

Keywords

Deduplication, Framework, Fdup

CNR authors

Manghi Paolo, De Bonis Michele, Atzori Claudio

CNR institutes

ISTI – Istituto di scienza e tecnologie dell'informazione "Alessandro Faedo"

ID: 476987

Year: 2022

Type: Articolo in rivista

Creation: 2023-01-26 15:41:45.000

Last update: 2023-01-27 10:05:10.000

External links

OAI-PMH: Dublin Core

OAI-PMH: Mods

OAI-PMH: RDF

DOI: 10.7717/PEERJ-CS.1058

URL: https://peerj.com/articles/cs-1058/

External IDs

CNR OAI-PMH: oai:it.cnr:prodotti:476987

DOI: 10.7717/PEERJ-CS.1058

Scopus: 2-s2.0-85143878034

ISI Web of Science (WOS): 000854914600001