An Efficient Duplicate Detection Algorithm Using Data Cleansing - Selvi, J; Gayathri, R (International Journal of Computer Sciences and Engineering, 2025)
The aim of the technique is to minimize the data duplication in the web mining patterns during the time of web based search in large data mining applications. Although there is a long line of work on identifying duplicates in relational data, only a few solutions focus on duplicate detection in more complex hierarchical structures, like XML data. In this system present a novel method for XML duplicate detection, called XML Dup. XML Dup uses a Bayesian network to determine the probability of two XML elements being duplicates, considering not only the information within the elements, but also the way that information is structured. In addition, to improve the efficiency of the network evaluation, a novel pruning strategy, capable of significant gains over the un optimized version of the algorithm, is presented. Through experiments, we show that our algorithm is able to achieve high precision and recall scores in several data sets. XML Dup is also able to outperform another state-of-the-art duplicate detection solution, both in terms of efficiency and of effectiveness.