An automatic identification of function words in TDIL tagged Bengali corpus - Pan, Subrata; Saha, Diganta (International Journal of Computer Sciences and Engineering, 2025)
Function words are quite high in textual information as compared to content words; where dimensionality is a critical challenge. Performance of text processing task deteriorates due to the presence of the function words in textual context. So, elimination of these words is an important activity in text processing to reduce the computational complexity and improve accuracy in the system. Many researches are performed for standard function words identification for English, Arabic, Chinese, Punjabi, Hindi, etc. In Bengali language processing, a limited number of standard function words are available. To address this limitation, we propose a computer based automatic system for identification of high scored function words from TDIL tagged Bengali corpus, Govt. of India. Total corpus consists of total 670,831 words and 134,884 distinct words. Our proposed system identifies 8 set of function words i.e. total 33,985 function words are identified in Literature domain of monolingual tagged corpus. At the end of our experiment, we achieved 290 standard function words as per their computed rank.