An Approach to Design Personalized Focused Crawler - Trivedi, Hardik P; Daxini, Gaurav N; Oswal, Jignesh A; Gor, Vinay D; Mali, Swati (International Journal of Computer Sciences and Engineering, 2014)
The amount of data and its dynamicity makes it impossible to crawl the World Wide Web (WWW) completely. Itï¿½s a challenge in front of crawlers to crawl only the relevant pages from this information explosion. Thus a focused crawler solves this issue of relevancy to a certain level, by focusing on web pages for some given topic or a set of topics. Also a focused crawler with a page change detection policy can help in narrowing down the search to only newer pages, and thus eliminates risk of redundancy and missing updated data. This paper proposes a policy for design of a focused crawler with web page change detection policy.