Text Similarity on Native Languages Documents - Kaur, Ramandeep; Kaur, Prabhjeet (International Journal of Computer Sciences and Engineering, 2021)
Text similarity of text measuring is a challenging task when text is in local languages and large in amount. Text measuring tools are easily available in the market but for regional languages very few tools are available. To figure out we have introduced a text similarity in native languages. In this paper, we are highlighting the Punjabi language where we find out that cosine similarity measures the accuracy of the Punjabi documents with other Punjabi documents. Text in both documents is divided into n-grams and then the common n-grams are found. The text in the documents is subject to pre-processing, which includes tokenization and punctuation removal, followed by stop words removal and stemming. After the preprocessing step, the similarity score is calculated using the cosine similarity. The purpose of doing this is to one step toward highlighting native languages. The features, performance, advantages, and disadvantages of various similarity measures are discussed. In this paper, we provide an efficient evaluation of all these measures and help the researchers to select the best measure according to their requirement.