Web Crawler : Review of Different Types of Web Crawler, Its Issues, Applications and Research Opportunities

• DOI: 10.26483/ijarcs.v8i3.3190
• Publication Date: 2017-01-01
• Authors: Keyur Desai; Virala Devulapalli; Smita Agrawal; Preeti Kathiria; Atul Patel
• Keywords: Web crawler; Focused crawler; Crawling

Today's search engines are equipped with dedicated agents known as “web crawlers” keen to crawling large web contents online which analyzed and indexed make the content available users. Crawlers act together thousands of servers over periods expanding from weeks several years. These crawlers visits pages every second, includes a high-performance fault manager, platform independent or dependent able get used wide range configurations without including additional hardware. This paper is focused on prerequisites crawler, process different types crawlers. give review about some potential issues related applications research area crawler.

Keywords: engine, www, Indexing, website analysis,