Open Access

ARTICLE

SmartCrawler: A Three-Stage Ranking Based Web Crawler for Harvesting Hidden Web Sources

Sawroop Kaur¹, Aman Singh^1,*, G. Geetha², Mehedi Masud³, Mohammed A. Alzain⁴

1 Computer Science and Engineering, Lovely Professional University, 144411, Punjab, India
2 CEO, Advanced Computing Research Society, Tamil Nadu, India
3 Department of Computer Science, Taif University, Taif, 21944, Saudi Arabia
4 Department of Information Technology, College of Computer and Information Technology, Taif University, Taif, 21944, Saudi Arabia

* Corresponding Author: Aman Singh. Email:

Computers, Materials & Continua 2021, 69(3), 2933-2948. https://doi.org/10.32604/cmc.2021.019030

Received 29 March 2021; Accepted 30 April 2021; Issue published 24 August 2021

Abstract

Web crawlers have evolved from performing a meagre task of collecting statistics, security testing, web indexing and numerous other examples. The size and dynamism of the web are making crawling an interesting and challenging task. Researchers have tackled various issues and challenges related to web crawling. One such issue is efficiently discovering hidden web data. Web crawler’s inability to work with form-based data, lack of benchmarks and standards for both performance measures and datasets for evaluation of the web crawlers make it still an immature research domain. The applications like vertical portals and data integration require hidden web crawling. Most of the existing methods are based on returning top k matches that makes exhaustive crawling difficult. The documents which are ranked high will be returned multiple times. The low ranked documents have slim chances of being retrieved. Discovering the hidden web sources and ranking them based on relevance is a core component of hidden web crawlers. The problem of ranking bias, heuristic approach and saturation of ranking algorithm led to low coverage. This research represents an enhanced ranking algorithm based on the triplet formula for prioritizing hidden websites to increase the coverage of the hidden web crawler.

---

Keywords

---