A study of different web-crawler behaviour

• DOI: 10.23919/fruct.2017.8071322
• Publication Date: 2017-10-25T19:29:28Z
• Authors: Alexander Menshch...; Antonina Komarova; Yurij Gatchin; Anatoly Korobeynikov; Nina Tishukova
• Keywords: Web crawler; Focused crawler; Crawling

The article deals with a study of web-crawler behaviour on different websites. A classification web-robots, information gathering tools and their detection methods are provided. Well-known scrapers analyzed the base large web-server log set. Experimental results demonstrate that web-robot can be distinguished from human by feature analysis. research used as basis for comprehensive intrusion prevention system development.