An efficient scheme to remove crawler traffic from the Internet

• DOI: 10.1109/icccn.2002.1043051
• Publication Date: 2003-06-25T23:36:53Z
• Authors: X. Yuan; M.H. MacGregor; J. Harms
• Keywords: Web crawler

We estimate that approximately 40% of current Internet traffic is due to Web crawlers retrieving pages for indexing. address this problem by introducing an efficient indexing system based on active networks. Our approach employs strategically placed routers constantly monitor passing traffic, analyze it, and then transmit the index data a dedicated back-end repository. simulations have shown up 30% more than crawler-based techniques.